2023-11-15T04:38:47.332 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-15T04:38:47.333 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --verbose --archive /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458408 --name yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458408/orig.config.yaml 2023-11-15T04:38:47.383 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-15T04:38:47.470 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458408 branch: wip-yuri4-testing-2023-11-13-0820-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7458408' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2023-11-13-0820-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 620f25c1794f62207424fc9319427943fb8d1fae ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 620f25c1794f62207424fc9319427943fb8d1fae selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-11-13-0820-pacific sha1: 620f25c1794f62207424fc9319427943fb8d1fae owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 4886 sha1: 620f25c1794f62207424fc9319427943fb8d1fae sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-11-13-0820-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 620f25c1794f62207424fc9319427943fb8d1fae targets: smithi163.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMbZUy/XzIHNhYEhMk6LE/09PqylObCPG64qcQlr3f87vYGqpGiLHv8nRzcVuIw3HMujdi66qU5CTxoX5cQFf50= smithi178.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNDbgYmv5wQAliGAJtcf6okb0phTjo29iPGnXDhDWh3Du70JD0vxr/tvfeVe/JRQu2Iacuq5Qf3GdbMyIlB8wmA= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-14_20:45:49 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3590714 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-11-15T04:38:47.470 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa; will attempt to use it 2023-11-15T04:38:47.471 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks 2023-11-15T04:38:47.472 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-15T04:38:47.473 INFO:teuthology.task.internal:Checking packages... 2023-11-15T04:38:47.497 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '620f25c1794f62207424fc9319427943fb8d1fae' 2023-11-15T04:38:47.498 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-15T04:38:47.498 INFO:teuthology.packaging:ref: None 2023-11-15T04:38:47.499 INFO:teuthology.packaging:tag: None 2023-11-15T04:38:47.499 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-11-13-0820-pacific 2023-11-15T04:38:47.499 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:38:47.499 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-11-13-0820-pacific 2023-11-15T04:38:47.640 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-170.g620f25c1 2023-11-15T04:38:47.640 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-15T04:38:47.646 INFO:teuthology.task.internal:no buildpackages task found 2023-11-15T04:38:47.646 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-15T04:38:47.664 INFO:teuthology.task.internal:Saving configuration 2023-11-15T04:38:47.687 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-15T04:38:47.692 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T04:38:47.719 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458408', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 04:32:51.442054', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMbZUy/XzIHNhYEhMk6LE/09PqylObCPG64qcQlr3f87vYGqpGiLHv8nRzcVuIw3HMujdi66qU5CTxoX5cQFf50='} 2023-11-15T04:38:47.740 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458408', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 04:32:51.443506', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNDbgYmv5wQAliGAJtcf6okb0phTjo29iPGnXDhDWh3Du70JD0vxr/tvfeVe/JRQu2Iacuq5Qf3GdbMyIlB8wmA='} 2023-11-15T04:38:47.740 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-15T04:38:47.746 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-11-15T04:38:47.746 INFO:teuthology.task.internal:roles: ubuntu@smithi178.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-11-15T04:38:47.749 INFO:teuthology.run_tasks:Running task console_log... 2023-11-15T04:38:47.820 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9bbbf5fb80>, signals=[15]) 2023-11-15T04:38:47.821 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-15T04:38:47.827 INFO:teuthology.task.internal:Opening connections... 2023-11-15T04:38:47.828 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi163.front.sepia.ceph.com 2023-11-15T04:38:47.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:38:47.899 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi178.front.sepia.ceph.com 2023-11-15T04:38:47.900 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:38:47.966 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-15T04:38:47.972 DEBUG:teuthology.orchestra.run.smithi163:> uname -m 2023-11-15T04:38:47.995 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64 2023-11-15T04:38:47.995 DEBUG:teuthology.orchestra.run.smithi163:> cat /etc/os-release 2023-11-15T04:38:48.056 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="CentOS Stream" 2023-11-15T04:38:48.056 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="8" 2023-11-15T04:38:48.057 INFO:teuthology.orchestra.run.smithi163.stdout:ID="centos" 2023-11-15T04:38:48.057 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="rhel fedora" 2023-11-15T04:38:48.057 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="8" 2023-11-15T04:38:48.057 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el8" 2023-11-15T04:38:48.057 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-15T04:38:48.058 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31" 2023-11-15T04:38:48.058 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-15T04:38:48.058 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://centos.org/" 2023-11-15T04:38:48.059 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-15T04:38:48.059 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-15T04:38:48.059 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-15T04:38:48.060 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2023-11-15T04:38:48.081 DEBUG:teuthology.orchestra.run.smithi178:> uname -m 2023-11-15T04:38:48.103 INFO:teuthology.orchestra.run.smithi178.stdout:x86_64 2023-11-15T04:38:48.104 DEBUG:teuthology.orchestra.run.smithi178:> cat /etc/os-release 2023-11-15T04:38:48.166 INFO:teuthology.orchestra.run.smithi178.stdout:NAME="CentOS Stream" 2023-11-15T04:38:48.166 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION="8" 2023-11-15T04:38:48.167 INFO:teuthology.orchestra.run.smithi178.stdout:ID="centos" 2023-11-15T04:38:48.167 INFO:teuthology.orchestra.run.smithi178.stdout:ID_LIKE="rhel fedora" 2023-11-15T04:38:48.167 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_ID="8" 2023-11-15T04:38:48.168 INFO:teuthology.orchestra.run.smithi178.stdout:PLATFORM_ID="platform:el8" 2023-11-15T04:38:48.168 INFO:teuthology.orchestra.run.smithi178.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-15T04:38:48.169 INFO:teuthology.orchestra.run.smithi178.stdout:ANSI_COLOR="0;31" 2023-11-15T04:38:48.169 INFO:teuthology.orchestra.run.smithi178.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-15T04:38:48.169 INFO:teuthology.orchestra.run.smithi178.stdout:HOME_URL="https://centos.org/" 2023-11-15T04:38:48.170 INFO:teuthology.orchestra.run.smithi178.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-15T04:38:48.170 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-15T04:38:48.171 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-15T04:38:48.171 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server 2023-11-15T04:38:48.197 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-15T04:38:48.207 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-15T04:38:48.212 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-15T04:38:48.213 DEBUG:teuthology.orchestra.run.smithi163:> test '!' -e /home/ubuntu/cephtest 2023-11-15T04:38:48.216 DEBUG:teuthology.orchestra.run.smithi178:> test '!' -e /home/ubuntu/cephtest 2023-11-15T04:38:48.237 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-15T04:38:48.244 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-15T04:38:48.244 DEBUG:teuthology.orchestra.run.smithi163:> test -z $(ls -A /var/lib/ceph) 2023-11-15T04:38:48.276 DEBUG:teuthology.orchestra.run.smithi178:> test -z $(ls -A /var/lib/ceph) 2023-11-15T04:38:48.308 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-15T04:38:48.364 INFO:teuthology.run_tasks:Running task kernel... 2023-11-15T04:38:48.381 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-15T04:38:48.382 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-15T04:38:48.382 DEBUG:teuthology.orchestra.run.smithi163:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-15T04:38:48.383 DEBUG:teuthology.orchestra.run.smithi178:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-15T04:38:48.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T04:38:48.404 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2023-11-15T04:38:48.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T04:38:48.406 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2023-11-15T04:38:48.466 INFO:teuthology.orchestra.run.smithi178.stdout:4.18.0-489.el8.x86_64 2023-11-15T04:38:48.466 INFO:teuthology.task.kernel:Running kernel on smithi178: 4.18.0-489.el8.x86_64 2023-11-15T04:38:48.467 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum install -y kernel 2023-11-15T04:38:48.468 INFO:teuthology.orchestra.run.smithi163.stdout:4.18.0-489.el8.x86_64 2023-11-15T04:38:48.468 INFO:teuthology.task.kernel:Running kernel on smithi163: 4.18.0-489.el8.x86_64 2023-11-15T04:38:48.468 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum install -y kernel 2023-11-15T04:38:53.116 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - AppStream 16 MB/s | 34 MB 00:02 2023-11-15T04:38:53.240 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - AppStream 16 MB/s | 34 MB 00:02 2023-11-15T04:39:00.333 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - BaseOS 51 MB/s | 53 MB 00:01 2023-11-15T04:39:00.498 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - BaseOS 52 MB/s | 53 MB 00:01 2023-11-15T04:39:08.524 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - Extras 29 kB/s | 18 kB 00:00 2023-11-15T04:39:08.631 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - Extras 31 kB/s | 18 kB 00:00 2023-11-15T04:39:09.382 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - Extras common packages 12 kB/s | 6.9 kB 00:00 2023-11-15T04:39:09.675 INFO:teuthology.orchestra.run.smithi163.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-11-15T04:39:10.770 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2023-11-15T04:39:12.449 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - Extras common packages 1.9 kB/s | 6.9 kB 00:03 2023-11-15T04:39:12.750 INFO:teuthology.orchestra.run.smithi178.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-11-15T04:39:14.676 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 540 kB/s | 24 kB 00:00 2023-11-15T04:39:18.283 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-15T04:39:18.284 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-15T04:39:18.570 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:39:18.571 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:39:18.571 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2023-11-15T04:39:18.572 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:39:18.572 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2023-11-15T04:39:18.572 INFO:teuthology.orchestra.run.smithi163.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-15T04:39:18.572 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2023-11-15T04:39:18.572 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-15T04:39:18.573 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-15T04:39:18.573 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:39:18.573 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2023-11-15T04:39:18.573 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:39:18.573 INFO:teuthology.orchestra.run.smithi163.stdout:Install 3 Packages 2023-11-15T04:39:18.574 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:39:18.574 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 89 M 2023-11-15T04:39:18.574 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 96 M 2023-11-15T04:39:18.575 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2023-11-15T04:39:19.044 INFO:teuthology.orchestra.run.smithi163.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 23 MB/s | 10 MB 00:00 2023-11-15T04:39:19.616 INFO:teuthology.orchestra.run.smithi163.stdout:(2/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 36 MB/s | 36 MB 00:01 2023-11-15T04:39:20.285 INFO:teuthology.orchestra.run.smithi163.stdout:(3/3): kernel-core-4.18.0-521.el8.x86_64.rpm 26 MB/s | 43 MB 00:01 2023-11-15T04:39:20.286 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:39:20.286 INFO:teuthology.orchestra.run.smithi163.stdout:Total 52 MB/s | 89 MB 00:01 2023-11-15T04:39:20.966 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2023-11-15T04:39:21.397 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2023-11-15T04:39:21.397 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2023-11-15T04:39:22.611 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2023-11-15T04:39:22.637 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2023-11-15T04:39:25.377 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2023-11-15T04:39:25.914 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T04:39:29.243 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 16 MB 00:16 2023-11-15T04:39:29.514 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T04:39:29.897 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T04:39:33.132 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 289 kB/s | 24 kB 00:00 2023-11-15T04:39:33.320 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T04:39:33.643 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:39:36.743 INFO:teuthology.orchestra.run.smithi178.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-15T04:39:36.744 INFO:teuthology.orchestra.run.smithi178.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-15T04:39:37.028 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:39:37.029 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:39:37.030 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2023-11-15T04:39:37.030 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:39:37.030 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-15T04:39:37.031 INFO:teuthology.orchestra.run.smithi178.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-15T04:39:37.031 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-11-15T04:39:37.031 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-15T04:39:37.031 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-15T04:39:37.031 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:39:37.032 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-15T04:39:37.032 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:39:37.032 INFO:teuthology.orchestra.run.smithi178.stdout:Install 3 Packages 2023-11-15T04:39:37.032 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:39:37.032 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 89 M 2023-11-15T04:39:37.033 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 96 M 2023-11-15T04:39:37.033 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-15T04:39:37.417 INFO:teuthology.orchestra.run.smithi178.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 29 MB/s | 10 MB 00:00 2023-11-15T04:39:38.208 INFO:teuthology.orchestra.run.smithi178.stdout:(2/3): kernel-core-4.18.0-521.el8.x86_64.rpm 38 MB/s | 43 MB 00:01 2023-11-15T04:39:38.817 INFO:teuthology.orchestra.run.smithi178.stdout:(3/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 21 MB/s | 36 MB 00:01 2023-11-15T04:39:38.818 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:39:38.818 INFO:teuthology.orchestra.run.smithi178.stdout:Total 50 MB/s | 89 MB 00:01 2023-11-15T04:39:39.496 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-15T04:39:39.926 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-15T04:39:39.926 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-15T04:39:41.161 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-15T04:39:41.181 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-15T04:39:43.914 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-15T04:39:44.452 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T04:39:48.001 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T04:39:48.409 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T04:39:51.844 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T04:39:52.141 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:39:55.698 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:39:55.721 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:39:56.066 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:39:56.067 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-15T04:39:56.067 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-15T04:39:56.905 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:39:56.905 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:39:56.906 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2023-11-15T04:39:56.906 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-15T04:39:56.906 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-15T04:39:56.906 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:39:56.907 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:39:57.044 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T04:39:57.304 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T04:39:57.305 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T04:39:57.305 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-15T04:39:57.305 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-15T04:39:57.305 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-15T04:39:57.306 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-11-15T04:39:57.306 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi163.front.sepia.ceph.com, path=None, version=distro) 2023-11-15T04:39:57.306 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T04:39:57.516 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T04:39:57.516 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T04:39:57.517 DEBUG:teuthology.orchestra.run.smithi163:> sudo rpm -qi grub2-tools 2023-11-15T04:39:57.551 INFO:teuthology.orchestra.run.smithi163.stdout:Name : grub2-tools 2023-11-15T04:39:57.551 INFO:teuthology.orchestra.run.smithi163.stdout:Epoch : 1 2023-11-15T04:39:57.551 INFO:teuthology.orchestra.run.smithi163.stdout:Version : 2.02 2023-11-15T04:39:57.551 INFO:teuthology.orchestra.run.smithi163.stdout:Release : 129.el8 2023-11-15T04:39:57.552 INFO:teuthology.orchestra.run.smithi163.stdout:Architecture: x86_64 2023-11-15T04:39:57.552 INFO:teuthology.orchestra.run.smithi163.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-15T04:39:57.552 INFO:teuthology.orchestra.run.smithi163.stdout:Group : System Environment/Base 2023-11-15T04:39:57.552 INFO:teuthology.orchestra.run.smithi163.stdout:Size : 9265586 2023-11-15T04:39:57.552 INFO:teuthology.orchestra.run.smithi163.stdout:License : GPLv3+ 2023-11-15T04:39:57.553 INFO:teuthology.orchestra.run.smithi163.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-15T04:39:57.553 INFO:teuthology.orchestra.run.smithi163.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-15T04:39:57.553 INFO:teuthology.orchestra.run.smithi163.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-15T04:39:57.553 INFO:teuthology.orchestra.run.smithi163.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-15T04:39:57.554 INFO:teuthology.orchestra.run.smithi163.stdout:Relocations : (not relocatable) 2023-11-15T04:39:57.554 INFO:teuthology.orchestra.run.smithi163.stdout:Packager : CentOS BuildSystem 2023-11-15T04:39:57.554 INFO:teuthology.orchestra.run.smithi163.stdout:Vendor : CentOS 2023-11-15T04:39:57.554 INFO:teuthology.orchestra.run.smithi163.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-15T04:39:57.555 INFO:teuthology.orchestra.run.smithi163.stdout:Summary : Support tools for GRUB. 2023-11-15T04:39:57.555 INFO:teuthology.orchestra.run.smithi163.stdout:Description : 2023-11-15T04:39:57.555 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:39:57.555 INFO:teuthology.orchestra.run.smithi163.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-15T04:39:57.555 INFO:teuthology.orchestra.run.smithi163.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-15T04:39:57.555 INFO:teuthology.orchestra.run.smithi163.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-15T04:39:57.556 INFO:teuthology.orchestra.run.smithi163.stdout:hardware devices. 2023-11-15T04:39:57.556 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:39:57.556 INFO:teuthology.orchestra.run.smithi163.stdout:This subpackage provides tools for support of all platforms. 2023-11-15T04:39:57.557 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-15T04:39:57.557 INFO:teuthology.task.kernel:Updating grub on smithi163 to boot 4.18.0-521.el8.x86_64 2023-11-15T04:39:57.558 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-15T04:39:59.279 INFO:teuthology.orchestra.run.smithi163.stderr:Generating grub configuration file ... 2023-11-15T04:40:01.573 INFO:teuthology.orchestra.run.smithi163.stderr:done 2023-11-15T04:40:01.575 DEBUG:teuthology.orchestra.run.smithi163:> mktemp 2023-11-15T04:40:01.602 INFO:teuthology.orchestra.run.smithi163.stdout:/tmp/tmp.7eGUhBtY6j 2023-11-15T04:40:01.602 DEBUG:teuthology.orchestra.run.smithi163:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.7eGUhBtY6j 2023-11-15T04:40:01.676 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0666 /tmp/tmp.7eGUhBtY6j 2023-11-15T04:40:01.820 DEBUG:teuthology.orchestra.remote:smithi163:/tmp/tmp.7eGUhBtY6j is 6KB 2023-11-15T04:40:01.869 DEBUG:teuthology.orchestra.run.smithi163:> rm -fr /tmp/tmp.7eGUhBtY6j 2023-11-15T04:40:01.887 DEBUG:teuthology.orchestra.run.smithi163:> sudo /bin/ls /boot/loader/entries || true 2023-11-15T04:40:01.959 INFO:teuthology.orchestra.run.smithi163.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-15T04:40:01.960 INFO:teuthology.orchestra.run.smithi163.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-15T04:40:01.960 INFO:teuthology.orchestra.run.smithi163.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-15T04:40:01.960 INFO:teuthology.orchestra.run.smithi163.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-15T04:40:01.961 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-15T04:40:02.111 DEBUG:teuthology.orchestra.run.smithi163:> sudo shutdown -r now 2023-11-15T04:40:13.978 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:40:14.001 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:40:14.347 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:40:14.347 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-15T04:40:14.348 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-15T04:40:15.149 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:40:15.150 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:40:15.150 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-15T04:40:15.150 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-15T04:40:15.150 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-15T04:40:15.151 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:40:15.151 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:40:15.342 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T04:40:15.560 INFO:teuthology.orchestra.run.smithi178.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T04:40:15.560 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T04:40:15.560 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-15T04:40:15.560 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-15T04:40:15.561 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-15T04:40:15.561 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-11-15T04:40:15.561 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi178.front.sepia.ceph.com, path=None, version=distro) 2023-11-15T04:40:15.561 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T04:40:15.770 INFO:teuthology.orchestra.run.smithi178.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T04:40:15.770 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T04:40:15.771 DEBUG:teuthology.orchestra.run.smithi178:> sudo rpm -qi grub2-tools 2023-11-15T04:40:15.805 INFO:teuthology.orchestra.run.smithi178.stdout:Name : grub2-tools 2023-11-15T04:40:15.805 INFO:teuthology.orchestra.run.smithi178.stdout:Epoch : 1 2023-11-15T04:40:15.805 INFO:teuthology.orchestra.run.smithi178.stdout:Version : 2.02 2023-11-15T04:40:15.805 INFO:teuthology.orchestra.run.smithi178.stdout:Release : 129.el8 2023-11-15T04:40:15.806 INFO:teuthology.orchestra.run.smithi178.stdout:Architecture: x86_64 2023-11-15T04:40:15.806 INFO:teuthology.orchestra.run.smithi178.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-15T04:40:15.806 INFO:teuthology.orchestra.run.smithi178.stdout:Group : System Environment/Base 2023-11-15T04:40:15.806 INFO:teuthology.orchestra.run.smithi178.stdout:Size : 9265586 2023-11-15T04:40:15.806 INFO:teuthology.orchestra.run.smithi178.stdout:License : GPLv3+ 2023-11-15T04:40:15.807 INFO:teuthology.orchestra.run.smithi178.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-15T04:40:15.807 INFO:teuthology.orchestra.run.smithi178.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-15T04:40:15.807 INFO:teuthology.orchestra.run.smithi178.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-15T04:40:15.807 INFO:teuthology.orchestra.run.smithi178.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-15T04:40:15.807 INFO:teuthology.orchestra.run.smithi178.stdout:Relocations : (not relocatable) 2023-11-15T04:40:15.808 INFO:teuthology.orchestra.run.smithi178.stdout:Packager : CentOS BuildSystem 2023-11-15T04:40:15.808 INFO:teuthology.orchestra.run.smithi178.stdout:Vendor : CentOS 2023-11-15T04:40:15.808 INFO:teuthology.orchestra.run.smithi178.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-15T04:40:15.808 INFO:teuthology.orchestra.run.smithi178.stdout:Summary : Support tools for GRUB. 2023-11-15T04:40:15.808 INFO:teuthology.orchestra.run.smithi178.stdout:Description : 2023-11-15T04:40:15.809 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:40:15.809 INFO:teuthology.orchestra.run.smithi178.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-15T04:40:15.809 INFO:teuthology.orchestra.run.smithi178.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-15T04:40:15.809 INFO:teuthology.orchestra.run.smithi178.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-15T04:40:15.809 INFO:teuthology.orchestra.run.smithi178.stdout:hardware devices. 2023-11-15T04:40:15.810 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:40:15.810 INFO:teuthology.orchestra.run.smithi178.stdout:This subpackage provides tools for support of all platforms. 2023-11-15T04:40:15.811 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-15T04:40:15.811 INFO:teuthology.task.kernel:Updating grub on smithi178 to boot 4.18.0-521.el8.x86_64 2023-11-15T04:40:15.812 DEBUG:teuthology.orchestra.run.smithi178:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-15T04:40:17.599 INFO:teuthology.orchestra.run.smithi178.stderr:Generating grub configuration file ... 2023-11-15T04:40:19.934 INFO:teuthology.orchestra.run.smithi178.stderr:done 2023-11-15T04:40:19.935 DEBUG:teuthology.orchestra.run.smithi178:> mktemp 2023-11-15T04:40:20.011 INFO:teuthology.orchestra.run.smithi178.stdout:/tmp/tmp.ZpGCC40Kff 2023-11-15T04:40:20.012 DEBUG:teuthology.orchestra.run.smithi178:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ZpGCC40Kff 2023-11-15T04:40:20.087 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0666 /tmp/tmp.ZpGCC40Kff 2023-11-15T04:40:20.224 DEBUG:teuthology.orchestra.remote:smithi178:/tmp/tmp.ZpGCC40Kff is 6KB 2023-11-15T04:40:20.273 DEBUG:teuthology.orchestra.run.smithi178:> rm -fr /tmp/tmp.ZpGCC40Kff 2023-11-15T04:40:20.292 DEBUG:teuthology.orchestra.run.smithi178:> sudo /bin/ls /boot/loader/entries || true 2023-11-15T04:40:20.361 INFO:teuthology.orchestra.run.smithi178.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-15T04:40:20.362 INFO:teuthology.orchestra.run.smithi178.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-15T04:40:20.362 INFO:teuthology.orchestra.run.smithi178.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-15T04:40:20.362 INFO:teuthology.orchestra.run.smithi178.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-15T04:40:20.363 DEBUG:teuthology.orchestra.run.smithi178:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-15T04:40:20.534 DEBUG:teuthology.orchestra.run.smithi178:> sudo shutdown -r now 2023-11-15T04:40:32.122 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-11-15T04:40:32.123 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-11-15T04:40:32.124 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:40:50.554 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-11-15T04:40:50.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2023-11-15T04:40:50.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:40:50.557 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2023-11-15T04:40:59.566 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-11-15T04:40:59.567 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:41:02.618 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2023-11-15T04:41:08.986 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2023-11-15T04:41:14.622 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-11-15T04:41:14.623 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:41:17.694 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2023-11-15T04:41:17.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2023-11-15T04:41:17.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:41:21.054 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2023-11-15T04:41:32.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-11-15T04:41:32.707 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:41:33.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2023-11-15T04:41:33.057 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:41:51.226 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2023-11-15T04:42:09.242 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-11-15T04:42:09.243 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:42:09.607 DEBUG:teuthology.orchestra.run.smithi163:> true 2023-11-15T04:42:09.950 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-11-15T04:42:09.951 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-521.el8.x86_64"... 2023-11-15T04:42:09.951 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2023-11-15T04:42:09.968 INFO:teuthology.orchestra.run.smithi163.stdout:4.18.0-521.el8.x86_64 2023-11-15T04:42:09.969 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-15T04:42:09.969 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-15T04:42:09.969 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-15T04:42:10.970 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-11-15T04:42:10.971 DEBUG:teuthology.orchestra.run.smithi163:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-15T04:42:11.046 INFO:teuthology.orchestra.run.smithi163.stdout:ttyS1 2023-11-15T04:42:11.062 DEBUG:teuthology.parallel:result is None 2023-11-15T04:42:33.059 DEBUG:teuthology.orchestra.remote:timed out 2023-11-15T04:42:48.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2023-11-15T04:42:48.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:42:48.347 DEBUG:teuthology.orchestra.run.smithi178:> true 2023-11-15T04:42:48.649 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2023-11-15T04:42:48.650 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-521.el8.x86_64"... 2023-11-15T04:42:48.650 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2023-11-15T04:42:48.669 INFO:teuthology.orchestra.run.smithi178.stdout:4.18.0-521.el8.x86_64 2023-11-15T04:42:48.670 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-15T04:42:48.670 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-15T04:42:48.670 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-15T04:42:49.671 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-11-15T04:42:49.672 DEBUG:teuthology.orchestra.run.smithi178:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-15T04:42:49.751 INFO:teuthology.orchestra.run.smithi178.stdout:ttyS1 2023-11-15T04:42:49.764 DEBUG:teuthology.parallel:result is None 2023-11-15T04:42:49.764 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-15T04:42:49.772 INFO:teuthology.task.internal:Creating test directory... 2023-11-15T04:42:49.773 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-15T04:42:49.777 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-15T04:42:49.798 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-15T04:42:49.805 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-15T04:42:49.838 INFO:teuthology.task.internal:Creating archive directory... 2023-11-15T04:42:49.838 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-15T04:42:49.841 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-15T04:42:49.916 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-15T04:42:49.923 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-15T04:42:49.924 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-15T04:42:49.927 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-15T04:42:49.965 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T04:42:49.976 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T04:42:49.978 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T04:42:49.989 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T04:42:49.991 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-15T04:42:49.998 INFO:teuthology.task.internal:Configuring sudo... 2023-11-15T04:42:49.998 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-15T04:42:50.001 DEBUG:teuthology.orchestra.run.smithi178:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-15T04:42:50.048 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-15T04:42:50.057 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-15T04:42:50.057 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-15T04:42:50.071 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-15T04:42:50.107 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T04:42:50.180 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T04:42:50.226 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T04:42:50.316 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T04:42:50.359 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-11-15T04:42:50.360 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-15T04:42:50.426 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T04:42:50.468 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T04:42:50.527 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T04:42:50.617 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T04:42:50.656 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-15T04:42:50.656 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-15T04:42:50.723 DEBUG:teuthology.orchestra.run.smithi163:> sudo service rsyslog restart 2023-11-15T04:42:50.726 DEBUG:teuthology.orchestra.run.smithi178:> sudo service rsyslog restart 2023-11-15T04:42:50.791 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T04:42:50.834 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T04:42:51.214 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-15T04:42:51.222 INFO:teuthology.task.internal:Starting timer... 2023-11-15T04:42:51.222 INFO:teuthology.run_tasks:Running task pcp... 2023-11-15T04:42:51.233 INFO:teuthology.run_tasks:Running task selinux... 2023-11-15T04:42:51.241 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-11-15T04:42:51.242 DEBUG:teuthology.orchestra.run.smithi163:> sudo service auditd rotate 2023-11-15T04:42:51.354 INFO:teuthology.orchestra.run.smithi163.stdout:Rotating logs: 2023-11-15T04:42:51.356 DEBUG:teuthology.orchestra.run.smithi178:> sudo service auditd rotate 2023-11-15T04:42:51.442 INFO:teuthology.orchestra.run.smithi178.stdout:Rotating logs: 2023-11-15T04:42:51.443 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-15T04:42:51.443 DEBUG:teuthology.orchestra.run.smithi163:> /usr/sbin/getenforce 2023-11-15T04:42:51.493 INFO:teuthology.orchestra.run.smithi163.stdout:Permissive 2023-11-15T04:42:51.494 DEBUG:teuthology.orchestra.run.smithi178:> /usr/sbin/getenforce 2023-11-15T04:42:51.548 INFO:teuthology.orchestra.run.smithi178.stdout:Permissive 2023-11-15T04:42:51.549 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi163.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi178.front.sepia.ceph.com': 'permissive'} 2023-11-15T04:42:51.549 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T04:42:51.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T04:42:51.583 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T04:42:51.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T04:42:51.611 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-15T04:42:51.611 DEBUG:teuthology.orchestra.run.smithi163:> sudo /usr/sbin/setenforce permissive 2023-11-15T04:42:51.653 DEBUG:teuthology.orchestra.run.smithi178:> sudo /usr/sbin/setenforce permissive 2023-11-15T04:42:51.688 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-15T04:42:51.697 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-15T04:42:51.698 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-15T04:42:51.716 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-15T04:42:51.718 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi163.front.sepia.ceph.com,smithi178.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-15T04:50:21.612 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi163.front.sepia.ceph.com'), Remote(name='ubuntu@smithi178.front.sepia.ceph.com')] 2023-11-15T04:50:21.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-11-15T04:50:21.615 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:50:21.682 DEBUG:teuthology.orchestra.run.smithi163:> true 2023-11-15T04:50:21.751 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-11-15T04:50:21.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2023-11-15T04:50:21.752 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:50:21.820 DEBUG:teuthology.orchestra.run.smithi178:> true 2023-11-15T04:50:21.889 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2023-11-15T04:50:21.890 INFO:teuthology.run_tasks:Running task clock... 2023-11-15T04:50:21.903 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-15T04:50:21.903 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-15T04:50:21.904 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T04:50:21.907 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-15T04:50:21.908 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T04:50:21.937 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-15T04:50:21.952 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-15T04:50:21.962 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-15T04:50:21.976 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-15T04:50:22.022 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: ntpd: command not found 2023-11-15T04:50:22.033 INFO:teuthology.orchestra.run.smithi163.stdout:506 Cannot talk to daemon 2023-11-15T04:50:22.044 INFO:teuthology.orchestra.run.smithi178.stderr:sudo: ntpd: command not found 2023-11-15T04:50:22.046 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-15T04:50:22.054 INFO:teuthology.orchestra.run.smithi178.stdout:506 Cannot talk to daemon 2023-11-15T04:50:22.060 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-15T04:50:22.068 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-15T04:50:22.081 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-15T04:50:22.132 INFO:teuthology.orchestra.run.smithi163.stderr:bash: ntpq: command not found 2023-11-15T04:50:22.135 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T04:50:22.136 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2023-11-15T04:50:22.136 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:50:22.136 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:50:22.136 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:50:22.136 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:50:22.149 INFO:teuthology.orchestra.run.smithi178.stderr:bash: ntpq: command not found 2023-11-15T04:50:22.152 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T04:50:22.153 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2023-11-15T04:50:22.153 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:50:22.153 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:50:22.153 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:50:22.153 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:50:22.154 INFO:teuthology.run_tasks:Running task pexec... 2023-11-15T04:50:22.164 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-15T04:50:22.165 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi163.front.sepia.ceph.com 2023-11-15T04:50:22.165 DEBUG:teuthology.orchestra.run.smithi163:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T04:50:22.167 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi178.front.sepia.ceph.com 2023-11-15T04:50:22.167 DEBUG:teuthology.orchestra.run.smithi178:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T04:50:23.060 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:39 ago on Wed 15 Nov 2023 04:48:43 AM UTC. 2023-11-15T04:50:23.077 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:40 ago on Wed 15 Nov 2023 04:48:42 AM UTC. 2023-11-15T04:50:25.174 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:50:25.175 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:50:25.175 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2023-11-15T04:50:25.175 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:50:25.175 INFO:teuthology.orchestra.run.smithi163.stdout:Resetting modules: 2023-11-15T04:50:25.175 INFO:teuthology.orchestra.run.smithi163.stdout: container-tools 2023-11-15T04:50:25.176 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:50:25.176 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2023-11-15T04:50:25.176 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:50:25.176 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:50:25.208 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:50:25.209 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:50:25.209 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2023-11-15T04:50:25.209 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:50:25.209 INFO:teuthology.orchestra.run.smithi178.stdout:Resetting modules: 2023-11-15T04:50:25.210 INFO:teuthology.orchestra.run.smithi178.stdout: container-tools 2023-11-15T04:50:25.210 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:50:25.210 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-15T04:50:25.210 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:50:25.210 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:50:25.435 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:50:25.487 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:50:26.379 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:43 ago on Wed 15 Nov 2023 04:48:43 AM UTC. 2023-11-15T04:50:26.433 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:44 ago on Wed 15 Nov 2023 04:48:42 AM UTC. 2023-11-15T04:50:28.479 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:50:28.536 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:50:28.656 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:50:28.656 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2023-11-15T04:50:28.657 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:50:28.657 INFO:teuthology.orchestra.run.smithi163.stdout:Installing group/module packages: 2023-11-15T04:50:28.657 INFO:teuthology.orchestra.run.smithi163.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-15T04:50:28.657 INFO:teuthology.orchestra.run.smithi163.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-15T04:50:28.658 INFO:teuthology.orchestra.run.smithi163.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-15T04:50:28.658 INFO:teuthology.orchestra.run.smithi163.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-15T04:50:28.658 INFO:teuthology.orchestra.run.smithi163.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-15T04:50:28.658 INFO:teuthology.orchestra.run.smithi163.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-15T04:50:28.658 INFO:teuthology.orchestra.run.smithi163.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-15T04:50:28.659 INFO:teuthology.orchestra.run.smithi163.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-15T04:50:28.659 INFO:teuthology.orchestra.run.smithi163.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-15T04:50:28.659 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2023-11-15T04:50:28.659 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-15T04:50:28.659 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-15T04:50:28.660 INFO:teuthology.orchestra.run.smithi163.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-15T04:50:28.660 INFO:teuthology.orchestra.run.smithi163.stdout:Installing weak dependencies: 2023-11-15T04:50:28.660 INFO:teuthology.orchestra.run.smithi163.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-15T04:50:28.660 INFO:teuthology.orchestra.run.smithi163.stdout:Installing module profiles: 2023-11-15T04:50:28.660 INFO:teuthology.orchestra.run.smithi163.stdout: container-tools/common 2023-11-15T04:50:28.661 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:50:28.661 INFO:teuthology.orchestra.run.smithi163.stdout:Enabling module streams: 2023-11-15T04:50:28.661 INFO:teuthology.orchestra.run.smithi163.stdout: container-tools rhel8 2023-11-15T04:50:28.661 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:50:28.661 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2023-11-15T04:50:28.662 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:50:28.662 INFO:teuthology.orchestra.run.smithi163.stdout:Install 13 Packages 2023-11-15T04:50:28.662 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:50:28.663 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 27 M 2023-11-15T04:50:28.663 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 89 M 2023-11-15T04:50:28.663 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2023-11-15T04:50:28.713 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:50:28.714 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2023-11-15T04:50:28.714 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:50:28.714 INFO:teuthology.orchestra.run.smithi178.stdout:Installing group/module packages: 2023-11-15T04:50:28.715 INFO:teuthology.orchestra.run.smithi178.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-15T04:50:28.715 INFO:teuthology.orchestra.run.smithi178.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-15T04:50:28.715 INFO:teuthology.orchestra.run.smithi178.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-15T04:50:28.715 INFO:teuthology.orchestra.run.smithi178.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-15T04:50:28.715 INFO:teuthology.orchestra.run.smithi178.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-15T04:50:28.716 INFO:teuthology.orchestra.run.smithi178.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-15T04:50:28.716 INFO:teuthology.orchestra.run.smithi178.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-15T04:50:28.716 INFO:teuthology.orchestra.run.smithi178.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-15T04:50:28.716 INFO:teuthology.orchestra.run.smithi178.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-15T04:50:28.716 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-11-15T04:50:28.716 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-15T04:50:28.717 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-15T04:50:28.717 INFO:teuthology.orchestra.run.smithi178.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-15T04:50:28.717 INFO:teuthology.orchestra.run.smithi178.stdout:Installing weak dependencies: 2023-11-15T04:50:28.717 INFO:teuthology.orchestra.run.smithi178.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-15T04:50:28.718 INFO:teuthology.orchestra.run.smithi178.stdout:Installing module profiles: 2023-11-15T04:50:28.718 INFO:teuthology.orchestra.run.smithi178.stdout: container-tools/common 2023-11-15T04:50:28.718 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:50:28.718 INFO:teuthology.orchestra.run.smithi178.stdout:Enabling module streams: 2023-11-15T04:50:28.718 INFO:teuthology.orchestra.run.smithi178.stdout: container-tools rhel8 2023-11-15T04:50:28.719 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:50:28.719 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-15T04:50:28.719 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:50:28.719 INFO:teuthology.orchestra.run.smithi178.stdout:Install 13 Packages 2023-11-15T04:50:28.719 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:50:28.720 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 27 M 2023-11-15T04:50:28.720 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 89 M 2023-11-15T04:50:28.721 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-15T04:50:29.136 INFO:teuthology.orchestra.run.smithi163.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 1.7 MB/s | 750 kB 00:00 2023-11-15T04:50:29.178 INFO:teuthology.orchestra.run.smithi163.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.1 MB/s | 1.0 MB 00:00 2023-11-15T04:50:29.191 INFO:teuthology.orchestra.run.smithi178.stdout:(1/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.3 MB/s | 1.0 MB 00:00 2023-11-15T04:50:29.203 INFO:teuthology.orchestra.run.smithi163.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 565 kB/s | 38 kB 00:00 2023-11-15T04:50:29.216 INFO:teuthology.orchestra.run.smithi178.stdout:(2/13): cockpit-podman-78-1.module_el8+712+4cd1 1.6 MB/s | 750 kB 00:00 2023-11-15T04:50:29.258 INFO:teuthology.orchestra.run.smithi178.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 566 kB/s | 38 kB 00:00 2023-11-15T04:50:29.261 INFO:teuthology.orchestra.run.smithi163.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.8 MB/s | 241 kB 00:00 2023-11-15T04:50:29.301 INFO:teuthology.orchestra.run.smithi178.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.8 MB/s | 241 kB 00:00 2023-11-15T04:50:29.362 INFO:teuthology.orchestra.run.smithi163.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.7 MB/s | 178 kB 00:00 2023-11-15T04:50:29.426 INFO:teuthology.orchestra.run.smithi178.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.4 MB/s | 178 kB 00:00 2023-11-15T04:50:29.462 INFO:teuthology.orchestra.run.smithi163.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 247 kB/s | 25 kB 00:00 2023-11-15T04:50:29.501 INFO:teuthology.orchestra.run.smithi178.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 330 kB/s | 25 kB 00:00 2023-11-15T04:50:29.568 INFO:teuthology.orchestra.run.smithi178.stdout:(7/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 13 MB/s | 3.9 MB 00:00 2023-11-15T04:50:29.593 INFO:teuthology.orchestra.run.smithi178.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.0 MB/s | 94 kB 00:00 2023-11-15T04:50:29.605 INFO:teuthology.orchestra.run.smithi163.stdout:(7/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 11 MB/s | 9.6 MB 00:00 2023-11-15T04:50:29.663 INFO:teuthology.orchestra.run.smithi163.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 8.5 MB/s | 3.9 MB 00:00 2023-11-15T04:50:29.689 INFO:teuthology.orchestra.run.smithi163.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 415 kB/s | 94 kB 00:00 2023-11-15T04:50:29.694 INFO:teuthology.orchestra.run.smithi178.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 10 MB/s | 9.6 MB 00:00 2023-11-15T04:50:29.794 INFO:teuthology.orchestra.run.smithi178.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 526 kB/s | 53 kB 00:00 2023-11-15T04:50:29.812 INFO:teuthology.orchestra.run.smithi163.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 427 kB/s | 53 kB 00:00 2023-11-15T04:50:29.844 INFO:teuthology.orchestra.run.smithi178.stdout:(11/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 10 MB/s | 2.5 MB 00:00 2023-11-15T04:50:29.861 INFO:teuthology.orchestra.run.smithi178.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 613 kB/s | 41 kB 00:00 2023-11-15T04:50:29.864 INFO:teuthology.orchestra.run.smithi163.stdout:(11/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 13 MB/s | 2.5 MB 00:00 2023-11-15T04:50:29.890 INFO:teuthology.orchestra.run.smithi163.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 532 kB/s | 41 kB 00:00 2023-11-15T04:50:29.987 INFO:teuthology.orchestra.run.smithi178.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 19 MB/s | 8.1 MB 00:00 2023-11-15T04:50:29.987 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:50:29.988 INFO:teuthology.orchestra.run.smithi178.stdout:Total 21 MB/s | 27 MB 00:01 2023-11-15T04:50:30.082 INFO:teuthology.orchestra.run.smithi163.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 17 MB/s | 8.1 MB 00:00 2023-11-15T04:50:30.082 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:50:30.083 INFO:teuthology.orchestra.run.smithi163.stdout:Total 19 MB/s | 27 MB 00:01 2023-11-15T04:50:30.254 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-15T04:50:30.275 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-15T04:50:30.276 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-15T04:50:30.351 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2023-11-15T04:50:30.373 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2023-11-15T04:50:30.373 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2023-11-15T04:50:30.550 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-15T04:50:30.553 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-15T04:50:30.636 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2023-11-15T04:50:30.639 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2023-11-15T04:50:31.261 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-15T04:50:31.438 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2023-11-15T04:50:31.462 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-15T04:50:31.657 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-15T04:50:31.709 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-15T04:50:31.929 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-15T04:50:31.949 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-15T04:50:32.126 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-15T04:50:32.172 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T04:50:32.389 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T04:50:32.741 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-15T04:50:32.972 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-15T04:50:33.732 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T04:50:33.975 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-15T04:50:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T04:50:34.201 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-15T04:50:34.282 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-15T04:50:34.461 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-15T04:50:34.516 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-15T04:50:34.816 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-15T04:50:34.868 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-15T04:50:35.242 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-15T04:50:35.782 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T04:50:36.033 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-15T04:50:36.177 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T04:50:36.221 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T04:50:36.630 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-15T04:50:36.671 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T04:50:36.671 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-15T04:50:36.671 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-15T04:50:36.672 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-15T04:50:36.672 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T04:50:36.672 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-15T04:50:36.672 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T04:50:36.672 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-15T04:50:36.673 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-15T04:50:36.673 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-15T04:50:36.673 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-15T04:50:36.673 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T04:50:36.673 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-15T04:50:36.861 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T04:50:37.330 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T04:50:37.330 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-15T04:50:37.330 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-15T04:50:37.330 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-15T04:50:37.331 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T04:50:37.331 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-15T04:50:37.331 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T04:50:37.331 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-15T04:50:37.331 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-15T04:50:37.332 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-15T04:50:37.332 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-15T04:50:37.332 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T04:50:37.332 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-15T04:50:37.592 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-15T04:50:37.592 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:50:37.592 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-15T04:50:37.593 INFO:teuthology.orchestra.run.smithi178.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:50:37.593 INFO:teuthology.orchestra.run.smithi178.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:50:37.593 INFO:teuthology.orchestra.run.smithi178.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-15T04:50:37.593 INFO:teuthology.orchestra.run.smithi178.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-15T04:50:37.593 INFO:teuthology.orchestra.run.smithi178.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:50:37.594 INFO:teuthology.orchestra.run.smithi178.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:50:37.594 INFO:teuthology.orchestra.run.smithi178.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-15T04:50:37.594 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-15T04:50:37.594 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-15T04:50:37.594 INFO:teuthology.orchestra.run.smithi178.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:50:37.595 INFO:teuthology.orchestra.run.smithi178.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:50:37.595 INFO:teuthology.orchestra.run.smithi178.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-15T04:50:37.595 INFO:teuthology.orchestra.run.smithi178.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-15T04:50:37.595 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:50:37.595 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:50:37.810 DEBUG:teuthology.parallel:result is None 2023-11-15T04:50:38.177 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-15T04:50:38.177 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:50:38.178 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2023-11-15T04:50:38.178 INFO:teuthology.orchestra.run.smithi163.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:50:38.178 INFO:teuthology.orchestra.run.smithi163.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:50:38.178 INFO:teuthology.orchestra.run.smithi163.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-15T04:50:38.179 INFO:teuthology.orchestra.run.smithi163.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-15T04:50:38.179 INFO:teuthology.orchestra.run.smithi163.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:50:38.179 INFO:teuthology.orchestra.run.smithi163.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:50:38.179 INFO:teuthology.orchestra.run.smithi163.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-15T04:50:38.179 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-15T04:50:38.179 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-15T04:50:38.180 INFO:teuthology.orchestra.run.smithi163.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:50:38.180 INFO:teuthology.orchestra.run.smithi163.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:50:38.180 INFO:teuthology.orchestra.run.smithi163.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-15T04:50:38.180 INFO:teuthology.orchestra.run.smithi163.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-15T04:50:38.180 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:50:38.181 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:50:38.442 DEBUG:teuthology.parallel:result is None 2023-11-15T04:50:38.443 INFO:teuthology.run_tasks:Running task pexec... 2023-11-15T04:50:38.452 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-15T04:50:38.453 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi163.front.sepia.ceph.com 2023-11-15T04:50:38.454 DEBUG:teuthology.orchestra.run.smithi163:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T04:50:38.454 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi178.front.sepia.ceph.com 2023-11-15T04:50:38.455 DEBUG:teuthology.orchestra.run.smithi178:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T04:50:38.616 DEBUG:teuthology.parallel:result is None 2023-11-15T04:50:38.653 DEBUG:teuthology.parallel:result is None 2023-11-15T04:50:38.653 INFO:teuthology.run_tasks:Running task install... 2023-11-15T04:50:38.661 DEBUG:teuthology.task.install:project ceph 2023-11-15T04:50:38.662 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'}} 2023-11-15T04:50:38.666 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'} 2023-11-15T04:50:38.667 INFO:teuthology.task.install:Using flavor: default 2023-11-15T04:50:38.673 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-15T04:50:38.673 INFO:teuthology.task.install:extra packages: [] 2023-11-15T04:50:38.674 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-15T04:50:38.675 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:50:38.675 INFO:teuthology.packaging:ref: None 2023-11-15T04:50:38.675 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:50:38.675 INFO:teuthology.packaging:branch: None 2023-11-15T04:50:38.676 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:50:38.863 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:50:38.864 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:50:38.866 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-15T04:50:38.867 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:50:38.867 INFO:teuthology.packaging:ref: None 2023-11-15T04:50:38.867 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:50:38.867 INFO:teuthology.packaging:branch: None 2023-11-15T04:50:38.868 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:50:39.067 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:50:39.068 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:50:39.290 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-15T04:50:39.291 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-15T04:50:39.295 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-15T04:50:39.296 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-15T04:50:39.341 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-15T04:50:39.342 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-11-15T04:50:39.342 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-15T04:50:39.348 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-15T04:50:39.348 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-15T04:50:39.349 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-15T04:50:39.374 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-15T04:50:39.375 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:50:39.375 INFO:teuthology.packaging:ref: None 2023-11-15T04:50:39.375 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:50:39.375 INFO:teuthology.packaging:branch: None 2023-11-15T04:50:39.376 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:50:39.376 DEBUG:teuthology.orchestra.run.smithi163:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-15T04:50:39.378 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-15T04:50:39.379 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:50:39.379 INFO:teuthology.packaging:ref: None 2023-11-15T04:50:39.379 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:50:39.379 INFO:teuthology.packaging:branch: None 2023-11-15T04:50:39.380 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:50:39.380 DEBUG:teuthology.orchestra.run.smithi178:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-15T04:50:39.444 DEBUG:teuthology.orchestra.run.smithi163:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-15T04:50:39.448 DEBUG:teuthology.orchestra.run.smithi178:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-15T04:50:39.522 DEBUG:teuthology.orchestra.run.smithi163:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-15T04:50:39.526 DEBUG:teuthology.orchestra.run.smithi178:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-15T04:50:39.589 INFO:teuthology.orchestra.run.smithi163.stdout:check_obsoletes = 1 2023-11-15T04:50:39.590 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum clean all 2023-11-15T04:50:39.593 INFO:teuthology.orchestra.run.smithi178.stdout:check_obsoletes = 1 2023-11-15T04:50:39.595 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum clean all 2023-11-15T04:50:39.860 INFO:teuthology.orchestra.run.smithi163.stdout:76 files removed 2023-11-15T04:50:39.876 INFO:teuthology.orchestra.run.smithi178.stdout:76 files removed 2023-11-15T04:50:39.888 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-radosgw 2023-11-15T04:50:39.904 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-radosgw 2023-11-15T04:50:40.377 INFO:teuthology.orchestra.run.smithi163.stdout:ceph packages for x86_64 891 kB/s | 74 kB 00:00 2023-11-15T04:50:40.394 INFO:teuthology.orchestra.run.smithi178.stdout:ceph packages for x86_64 886 kB/s | 74 kB 00:00 2023-11-15T04:50:40.494 INFO:teuthology.orchestra.run.smithi163.stdout:ceph noarch packages 189 kB/s | 15 kB 00:00 2023-11-15T04:50:40.503 INFO:teuthology.orchestra.run.smithi178.stdout:ceph noarch packages 213 kB/s | 15 kB 00:00 2023-11-15T04:50:40.595 INFO:teuthology.orchestra.run.smithi163.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2023-11-15T04:50:40.621 INFO:teuthology.orchestra.run.smithi178.stdout:ceph source packages 20 kB/s | 1.7 kB 00:00 2023-11-15T04:50:42.190 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - AppStream 22 MB/s | 34 MB 00:01 2023-11-15T04:50:42.330 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - AppStream 20 MB/s | 34 MB 00:01 2023-11-15T04:50:49.764 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - BaseOS 38 MB/s | 53 MB 00:01 2023-11-15T04:50:50.547 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - BaseOS 26 MB/s | 53 MB 00:02 2023-11-15T04:50:57.807 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - Extras 38 kB/s | 18 kB 00:00 2023-11-15T04:50:58.480 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - Extras common packages 14 kB/s | 6.9 kB 00:00 2023-11-15T04:50:58.705 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - Extras 30 kB/s | 18 kB 00:00 2023-11-15T04:50:59.089 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - PowerTools 14 MB/s | 6.6 MB 00:00 2023-11-15T04:50:59.397 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.9 kB 00:00 2023-11-15T04:51:00.157 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - PowerTools 11 MB/s | 6.6 MB 00:00 2023-11-15T04:51:00.509 INFO:teuthology.orchestra.run.smithi178.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-11-15T04:51:01.609 INFO:teuthology.orchestra.run.smithi163.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-11-15T04:51:02.269 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 9.9 MB/s | 16 MB 00:01 2023-11-15T04:51:02.768 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:01 2023-11-15T04:51:06.201 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 274 kB/s | 24 kB 00:00 2023-11-15T04:51:06.651 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 321 kB/s | 24 kB 00:00 2023-11-15T04:51:10.864 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:51:10.867 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:51:10.867 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2023-11-15T04:51:10.867 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:51:10.868 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-15T04:51:10.868 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-15T04:51:10.868 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrading: 2023-11-15T04:51:10.868 INFO:teuthology.orchestra.run.smithi178.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T04:51:10.868 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-15T04:51:10.869 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-11-15T04:51:10.869 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-15T04:51:10.869 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-15T04:51:10.869 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-15T04:51:10.869 INFO:teuthology.orchestra.run.smithi178.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-15T04:51:10.870 INFO:teuthology.orchestra.run.smithi178.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-15T04:51:10.870 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-15T04:51:10.870 INFO:teuthology.orchestra.run.smithi178.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-15T04:51:10.870 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-15T04:51:10.871 INFO:teuthology.orchestra.run.smithi178.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-15T04:51:10.871 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-15T04:51:10.871 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-15T04:51:10.871 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-15T04:51:10.871 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-15T04:51:10.871 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-15T04:51:10.872 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-15T04:51:10.872 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-15T04:51:10.872 INFO:teuthology.orchestra.run.smithi178.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-15T04:51:10.872 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-15T04:51:10.872 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-15T04:51:10.873 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-15T04:51:10.873 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-15T04:51:10.873 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-15T04:51:10.873 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-15T04:51:10.873 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-15T04:51:10.874 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:51:10.874 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-15T04:51:10.874 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:51:10.874 INFO:teuthology.orchestra.run.smithi178.stdout:Install 25 Packages 2023-11-15T04:51:10.874 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrade 2 Packages 2023-11-15T04:51:10.875 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:51:10.876 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 59 M 2023-11-15T04:51:10.876 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-15T04:51:11.289 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:51:11.292 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:51:11.292 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2023-11-15T04:51:11.292 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:51:11.293 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2023-11-15T04:51:11.293 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-15T04:51:11.293 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrading: 2023-11-15T04:51:11.294 INFO:teuthology.orchestra.run.smithi163.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T04:51:11.294 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-15T04:51:11.294 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2023-11-15T04:51:11.295 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-15T04:51:11.295 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-15T04:51:11.296 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-15T04:51:11.296 INFO:teuthology.orchestra.run.smithi163.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-15T04:51:11.296 INFO:teuthology.orchestra.run.smithi163.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-15T04:51:11.297 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-15T04:51:11.297 INFO:teuthology.orchestra.run.smithi163.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-15T04:51:11.298 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-15T04:51:11.298 INFO:teuthology.orchestra.run.smithi163.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-15T04:51:11.298 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-15T04:51:11.299 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-15T04:51:11.299 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-15T04:51:11.299 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-15T04:51:11.300 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-15T04:51:11.300 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-15T04:51:11.300 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-15T04:51:11.301 INFO:teuthology.orchestra.run.smithi163.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-15T04:51:11.301 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-15T04:51:11.301 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-15T04:51:11.302 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-15T04:51:11.302 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-15T04:51:11.303 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-15T04:51:11.303 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-15T04:51:11.303 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-15T04:51:11.303 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:51:11.304 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2023-11-15T04:51:11.304 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:51:11.304 INFO:teuthology.orchestra.run.smithi163.stdout:Install 25 Packages 2023-11-15T04:51:11.304 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrade 2 Packages 2023-11-15T04:51:11.304 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:51:11.309 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 59 M 2023-11-15T04:51:11.309 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2023-11-15T04:51:12.460 INFO:teuthology.orchestra.run.smithi178.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 37 MB/s | 6.0 MB 00:00 2023-11-15T04:51:12.485 INFO:teuthology.orchestra.run.smithi178.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 999 kB/s | 25 kB 00:00 2023-11-15T04:51:12.519 INFO:teuthology.orchestra.run.smithi178.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 23 MB/s | 801 kB 00:00 2023-11-15T04:51:12.561 INFO:teuthology.orchestra.run.smithi178.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 11 MB/s | 486 kB 00:00 2023-11-15T04:51:12.710 INFO:teuthology.orchestra.run.smithi178.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 27 MB/s | 4.1 MB 00:00 2023-11-15T04:51:12.745 INFO:teuthology.orchestra.run.smithi178.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.3 MB/s | 44 kB 00:00 2023-11-15T04:51:12.770 INFO:teuthology.orchestra.run.smithi178.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-11-15T04:51:12.795 INFO:teuthology.orchestra.run.smithi178.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2023-11-15T04:51:12.837 INFO:teuthology.orchestra.run.smithi178.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 9.1 MB/s | 387 kB 00:00 2023-11-15T04:51:12.871 INFO:teuthology.orchestra.run.smithi178.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2023-11-15T04:51:12.896 INFO:teuthology.orchestra.run.smithi178.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-11-15T04:51:13.039 INFO:teuthology.orchestra.run.smithi178.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 17 MB/s | 13 MB 00:00 2023-11-15T04:51:13.128 INFO:teuthology.orchestra.run.smithi163.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 28 MB/s | 6.0 MB 00:00 2023-11-15T04:51:13.154 INFO:teuthology.orchestra.run.smithi163.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2023-11-15T04:51:13.196 INFO:teuthology.orchestra.run.smithi163.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2023-11-15T04:51:13.229 INFO:teuthology.orchestra.run.smithi163.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-11-15T04:51:13.314 INFO:teuthology.orchestra.run.smithi178.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:01 2023-11-15T04:51:13.330 INFO:teuthology.orchestra.run.smithi163.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2023-11-15T04:51:13.339 INFO:teuthology.orchestra.run.smithi178.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 328 kB/s | 145 kB 00:00 2023-11-15T04:51:13.355 INFO:teuthology.orchestra.run.smithi163.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2023-11-15T04:51:13.380 INFO:teuthology.orchestra.run.smithi163.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-11-15T04:51:13.406 INFO:teuthology.orchestra.run.smithi163.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2023-11-15T04:51:13.440 INFO:teuthology.orchestra.run.smithi163.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-11-15T04:51:13.464 INFO:teuthology.orchestra.run.smithi178.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 351 kB/s | 44 kB 00:00 2023-11-15T04:51:13.473 INFO:teuthology.orchestra.run.smithi163.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2023-11-15T04:51:13.556 INFO:teuthology.orchestra.run.smithi178.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 462 kB/s | 42 kB 00:00 2023-11-15T04:51:13.607 INFO:teuthology.orchestra.run.smithi163.stdout:(11/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 18 MB/s | 13 MB 00:00 2023-11-15T04:51:13.615 INFO:teuthology.orchestra.run.smithi178.stdout:(17/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 806 kB/s | 47 kB 00:00 2023-11-15T04:51:13.633 INFO:teuthology.orchestra.run.smithi163.stdout:(12/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 719 kB/s | 114 kB 00:00 2023-11-15T04:51:13.657 INFO:teuthology.orchestra.run.smithi178.stdout:(18/27): ndctl-libs-71.1-7.el8.x86_64.rpm 1.9 MB/s | 80 kB 00:00 2023-11-15T04:51:13.765 INFO:teuthology.orchestra.run.smithi178.stdout:(19/27): librdkafka-0.11.4-3.el8.x86_64.rpm 487 kB/s | 354 kB 00:00 2023-11-15T04:51:13.832 INFO:teuthology.orchestra.run.smithi178.stdout:(20/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 501 kB/s | 259 kB 00:00 2023-11-15T04:51:13.916 INFO:teuthology.orchestra.run.smithi163.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:01 2023-11-15T04:51:14.199 INFO:teuthology.orchestra.run.smithi178.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 214 kB/s | 116 kB 00:00 2023-11-15T04:51:14.275 INFO:teuthology.orchestra.run.smithi163.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 218 kB/s | 145 kB 00:00 2023-11-15T04:51:14.367 INFO:teuthology.orchestra.run.smithi163.stdout:(15/27): librdkafka-0.11.4-3.el8.x86_64.rpm 482 kB/s | 354 kB 00:00 2023-11-15T04:51:14.392 INFO:teuthology.orchestra.run.smithi163.stdout:(16/27): python3-prettytable-0.7.2-14.el8.noarc 376 kB/s | 44 kB 00:00 2023-11-15T04:51:14.408 INFO:teuthology.orchestra.run.smithi178.stdout:(22/27): liboath-2.6.2-3.el8.x86_64.rpm 282 kB/s | 59 kB 00:00 2023-11-15T04:51:14.433 INFO:teuthology.orchestra.run.smithi178.stdout:(23/27): leveldb-1.22-1.el8.x86_64.rpm 301 kB/s | 181 kB 00:00 2023-11-15T04:51:14.439 INFO:teuthology.orchestra.run.smithi163.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 497 kB/s | 259 kB 00:00 2023-11-15T04:51:14.518 INFO:teuthology.orchestra.run.smithi163.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 280 kB/s | 42 kB 00:00 2023-11-15T04:51:14.534 INFO:teuthology.orchestra.run.smithi178.stdout:(24/27): librados2-16.2.4-0.el8.x86_64.rpm 37 MB/s | 3.7 MB 00:00 2023-11-15T04:51:14.543 INFO:teuthology.orchestra.run.smithi163.stdout:(19/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 312 kB/s | 47 kB 00:00 2023-11-15T04:51:14.567 INFO:teuthology.orchestra.run.smithi178.stdout:(25/27): gperftools-libs-2.7-9.el8.x86_64.rpm 382 kB/s | 306 kB 00:00 2023-11-15T04:51:14.568 INFO:teuthology.orchestra.run.smithi163.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 621 kB/s | 80 kB 00:00 2023-11-15T04:51:14.592 INFO:teuthology.orchestra.run.smithi178.stdout:(26/27): libunwind-1.3.1-3.el8.x86_64.rpm 407 kB/s | 75 kB 00:00 2023-11-15T04:51:14.676 INFO:teuthology.orchestra.run.smithi178.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 29 MB/s | 4.0 MB 00:00 2023-11-15T04:51:14.677 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:51:14.678 INFO:teuthology.orchestra.run.smithi178.stdout:Total 15 MB/s | 59 MB 00:03 2023-11-15T04:51:14.735 INFO:teuthology.orchestra.run.smithi163.stdout:(21/27): leveldb-1.22-1.el8.x86_64.rpm 1.1 MB/s | 181 kB 00:00 2023-11-15T04:51:14.745 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-15T04:51:14.761 INFO:teuthology.orchestra.run.smithi163.stdout:(22/27): fmt-6.2.1-1.el8.x86_64.rpm 535 kB/s | 116 kB 00:00 2023-11-15T04:51:14.786 INFO:teuthology.orchestra.run.smithi163.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2023-11-15T04:51:14.793 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-15T04:51:14.793 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-15T04:51:14.802 INFO:teuthology.orchestra.run.smithi163.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 877 kB/s | 59 kB 00:00 2023-11-15T04:51:14.828 INFO:teuthology.orchestra.run.smithi163.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2023-11-15T04:51:14.953 INFO:teuthology.orchestra.run.smithi163.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 25 MB/s | 3.7 MB 00:00 2023-11-15T04:51:15.012 INFO:teuthology.orchestra.run.smithi163.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 19 MB/s | 4.0 MB 00:00 2023-11-15T04:51:15.016 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:51:15.017 INFO:teuthology.orchestra.run.smithi163.stdout:Total 16 MB/s | 59 MB 00:03 2023-11-15T04:51:15.083 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2023-11-15T04:51:15.130 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2023-11-15T04:51:15.131 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2023-11-15T04:51:15.155 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-15T04:51:15.158 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-15T04:51:15.475 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2023-11-15T04:51:15.479 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2023-11-15T04:51:16.296 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-15T04:51:16.518 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-15T04:51:16.712 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2023-11-15T04:51:16.764 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-15T04:51:16.939 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T04:51:16.945 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-15T04:51:17.193 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-15T04:51:17.351 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T04:51:17.396 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T04:51:17.573 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:51:17.651 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:51:17.847 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T04:51:17.928 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T04:51:18.098 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:51:18.188 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:51:18.346 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:51:18.379 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-15T04:51:18.450 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:51:18.581 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T04:51:18.739 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T04:51:18.896 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:51:18.993 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:51:19.004 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T04:51:19.179 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-15T04:51:19.180 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:51:19.304 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:51:19.363 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T04:51:19.720 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-15T04:51:19.802 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T04:51:19.914 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T04:51:19.986 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T04:51:20.012 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:51:20.097 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:51:20.186 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T04:51:20.301 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-15T04:51:20.398 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T04:51:20.521 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T04:51:20.589 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T04:51:20.590 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T04:51:20.663 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T04:51:20.793 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T04:51:20.839 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T04:51:20.996 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T04:51:21.196 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T04:51:21.260 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T04:51:21.274 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T04:51:21.462 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T04:51:21.497 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T04:51:21.583 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T04:51:21.776 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-15T04:51:21.894 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T04:51:22.040 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-15T04:51:22.137 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T04:51:22.238 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T04:51:22.288 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T04:51:22.362 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T04:51:22.442 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-15T04:51:22.576 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-15T04:51:22.697 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-15T04:51:22.788 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-15T04:51:22.870 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T04:51:22.947 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T04:51:22.979 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-15T04:51:23.179 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-15T04:51:23.180 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-15T04:51:23.395 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-15T04:51:23.636 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-15T04:51:23.887 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-15T04:51:25.759 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:51:26.030 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:51:26.454 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:51:26.720 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:51:26.724 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:51:27.036 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T04:51:27.148 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T04:51:27.321 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:51:27.414 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:51:27.776 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T04:51:27.886 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T04:51:28.061 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:51:45.625 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:51:45.625 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /sys 2023-11-15T04:51:45.625 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /proc 2023-11-15T04:51:45.626 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /dev 2023-11-15T04:51:45.626 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /run 2023-11-15T04:51:45.626 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /mnt 2023-11-15T04:51:45.626 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /var/tmp 2023-11-15T04:51:45.626 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /home 2023-11-15T04:51:45.627 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /tmp 2023-11-15T04:51:45.627 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /dev 2023-11-15T04:51:45.627 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:51:46.789 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:51:46.789 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /sys 2023-11-15T04:51:46.789 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /proc 2023-11-15T04:51:46.789 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /dev 2023-11-15T04:51:46.790 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /run 2023-11-15T04:51:46.790 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /mnt 2023-11-15T04:51:46.790 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /var/tmp 2023-11-15T04:51:46.790 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /home 2023-11-15T04:51:46.790 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /tmp 2023-11-15T04:51:46.791 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /dev 2023-11-15T04:51:46.791 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:51:46.957 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T04:51:47.034 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T04:51:47.035 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T04:51:47.286 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T04:51:47.287 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:51:48.026 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:51:48.027 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-15T04:51:48.027 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-15T04:51:48.027 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-15T04:51:48.027 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:51:48.028 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T04:51:48.028 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:51:48.028 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-15T04:51:48.028 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-15T04:51:48.028 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:51:48.029 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-15T04:51:48.029 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-15T04:51:48.029 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T04:51:48.029 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T04:51:48.029 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-15T04:51:48.030 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-15T04:51:48.030 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-15T04:51:48.030 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-15T04:51:48.030 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-15T04:51:48.030 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-15T04:51:48.031 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-15T04:51:48.031 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-15T04:51:48.031 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-15T04:51:48.031 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-15T04:51:48.031 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-15T04:51:48.032 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-15T04:51:48.032 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:51:48.032 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-15T04:51:48.032 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-15T04:51:48.154 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T04:51:48.238 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T04:51:48.238 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T04:51:48.468 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T04:51:48.469 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:51:49.182 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:51:49.183 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-15T04:51:49.183 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-15T04:51:49.183 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-15T04:51:49.183 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:51:49.183 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T04:51:49.184 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:51:49.184 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-15T04:51:49.184 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-15T04:51:49.185 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:51:49.185 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-15T04:51:49.185 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-15T04:51:49.185 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T04:51:49.185 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T04:51:49.186 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-15T04:51:49.186 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-15T04:51:49.186 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-15T04:51:49.186 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-15T04:51:49.187 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-15T04:51:49.187 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-15T04:51:49.187 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-15T04:51:49.187 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-15T04:51:49.187 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-15T04:51:49.188 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-15T04:51:49.188 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-15T04:51:49.188 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-15T04:51:49.188 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:51:49.189 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-15T04:51:49.189 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-15T04:51:49.229 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:51:49.230 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:51:49.230 INFO:teuthology.orchestra.run.smithi178.stdout:Upgraded: 2023-11-15T04:51:49.230 INFO:teuthology.orchestra.run.smithi178.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:49.230 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-15T04:51:49.230 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:49.231 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:49.231 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:49.231 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:49.231 INFO:teuthology.orchestra.run.smithi178.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-15T04:51:49.231 INFO:teuthology.orchestra.run.smithi178.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-15T04:51:49.232 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-15T04:51:49.232 INFO:teuthology.orchestra.run.smithi178.stdout: leveldb-1.22-1.el8.x86_64 2023-11-15T04:51:49.232 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:49.232 INFO:teuthology.orchestra.run.smithi178.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-15T04:51:49.232 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-15T04:51:49.233 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-15T04:51:49.233 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:49.233 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-15T04:51:49.234 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:49.234 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-15T04:51:49.234 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-15T04:51:49.234 INFO:teuthology.orchestra.run.smithi178.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-15T04:51:49.234 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:49.235 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:49.235 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:49.235 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-15T04:51:49.235 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:49.235 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:49.236 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:49.236 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:51:49.236 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:51:49.545 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-test 2023-11-15T04:51:50.383 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:51:50.383 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:51:50.384 INFO:teuthology.orchestra.run.smithi163.stdout:Upgraded: 2023-11-15T04:51:50.384 INFO:teuthology.orchestra.run.smithi163.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:50.384 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2023-11-15T04:51:50.384 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:50.385 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:50.385 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:50.385 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:50.385 INFO:teuthology.orchestra.run.smithi163.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-15T04:51:50.385 INFO:teuthology.orchestra.run.smithi163.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-15T04:51:50.386 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-15T04:51:50.386 INFO:teuthology.orchestra.run.smithi163.stdout: leveldb-1.22-1.el8.x86_64 2023-11-15T04:51:50.386 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:50.386 INFO:teuthology.orchestra.run.smithi163.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-15T04:51:50.386 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-15T04:51:50.387 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-15T04:51:50.387 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:50.387 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-15T04:51:50.387 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:50.387 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-15T04:51:50.388 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-15T04:51:50.388 INFO:teuthology.orchestra.run.smithi163.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-15T04:51:50.388 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:50.388 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:50.389 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:50.389 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-15T04:51:50.389 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:50.389 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:50.389 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-15T04:51:50.390 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:51:50.390 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:51:50.401 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:44 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:51:50.699 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-test 2023-11-15T04:51:51.558 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:45 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:51:52.020 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:51:52.021 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:51:52.021 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2023-11-15T04:51:52.021 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:51:52.021 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-15T04:51:52.022 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-15T04:51:52.022 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-11-15T04:51:52.022 INFO:teuthology.orchestra.run.smithi178.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-15T04:51:52.022 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-15T04:51:52.023 INFO:teuthology.orchestra.run.smithi178.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-15T04:51:52.023 INFO:teuthology.orchestra.run.smithi178.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-15T04:51:52.023 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-15T04:51:52.023 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:51:52.023 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-15T04:51:52.024 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:51:52.024 INFO:teuthology.orchestra.run.smithi178.stdout:Install 6 Packages 2023-11-15T04:51:52.024 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:51:52.025 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 53 M 2023-11-15T04:51:52.025 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 229 M 2023-11-15T04:51:52.026 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-15T04:51:52.313 INFO:teuthology.orchestra.run.smithi178.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 670 kB/s | 169 kB 00:00 2023-11-15T04:51:52.826 INFO:teuthology.orchestra.run.smithi178.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 68 MB/s | 52 MB 00:00 2023-11-15T04:51:52.851 INFO:teuthology.orchestra.run.smithi178.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 257 kB/s | 203 kB 00:00 2023-11-15T04:51:52.934 INFO:teuthology.orchestra.run.smithi178.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 848 kB/s | 71 kB 00:00 2023-11-15T04:51:53.093 INFO:teuthology.orchestra.run.smithi178.stdout:(5/6): oniguruma-6.8.2-2.el8.x86_64.rpm 240 kB/s | 187 kB 00:00 2023-11-15T04:51:53.185 INFO:teuthology.orchestra.run.smithi178.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 901 kB/s | 323 kB 00:00 2023-11-15T04:51:53.186 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:51:53.186 INFO:teuthology.orchestra.run.smithi178.stdout:Total 45 MB/s | 53 MB 00:01 2023-11-15T04:51:53.223 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:51:53.224 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:51:53.224 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2023-11-15T04:51:53.224 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:51:53.225 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2023-11-15T04:51:53.225 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-15T04:51:53.225 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2023-11-15T04:51:53.225 INFO:teuthology.orchestra.run.smithi163.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-15T04:51:53.225 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-15T04:51:53.226 INFO:teuthology.orchestra.run.smithi163.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-15T04:51:53.226 INFO:teuthology.orchestra.run.smithi163.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-15T04:51:53.226 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-15T04:51:53.226 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:51:53.226 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2023-11-15T04:51:53.227 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:51:53.227 INFO:teuthology.orchestra.run.smithi163.stdout:Install 6 Packages 2023-11-15T04:51:53.227 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:51:53.228 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 53 M 2023-11-15T04:51:53.228 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 229 M 2023-11-15T04:51:53.228 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2023-11-15T04:51:53.229 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-15T04:51:53.249 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-15T04:51:53.250 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-15T04:51:53.464 INFO:teuthology.orchestra.run.smithi163.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.6 MB/s | 169 kB 00:00 2023-11-15T04:51:53.555 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-15T04:51:53.558 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-15T04:51:54.026 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-15T04:51:54.198 INFO:teuthology.orchestra.run.smithi163.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 67 MB/s | 52 MB 00:00 2023-11-15T04:51:54.223 INFO:teuthology.orchestra.run.smithi163.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 252 kB/s | 203 kB 00:00 2023-11-15T04:51:54.256 INFO:teuthology.orchestra.run.smithi163.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 236 kB/s | 187 kB 00:00 2023-11-15T04:51:54.302 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-15T04:51:54.307 INFO:teuthology.orchestra.run.smithi163.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 844 kB/s | 71 kB 00:00 2023-11-15T04:51:54.493 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-15T04:51:54.565 INFO:teuthology.orchestra.run.smithi163.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 880 kB/s | 323 kB 00:00 2023-11-15T04:51:54.566 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:51:54.567 INFO:teuthology.orchestra.run.smithi163.stdout:Total 39 MB/s | 53 MB 00:01 2023-11-15T04:51:54.609 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2023-11-15T04:51:54.631 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2023-11-15T04:51:54.631 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2023-11-15T04:51:54.699 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T04:51:54.764 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T04:51:54.939 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-15T04:51:54.943 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2023-11-15T04:51:54.945 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2023-11-15T04:51:55.107 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T04:51:55.451 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2023-11-15T04:51:55.709 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-15T04:51:55.917 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-15T04:51:56.106 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T04:51:56.186 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T04:51:56.388 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-15T04:51:56.565 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T04:51:59.936 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T04:52:00.213 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T04:52:00.492 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T04:52:00.493 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-15T04:52:00.493 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-15T04:52:00.493 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-15T04:52:00.493 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-15T04:52:00.493 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-15T04:52:01.201 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-15T04:52:01.201 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:01.201 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-15T04:52:01.202 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-15T04:52:01.202 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-15T04:52:01.202 INFO:teuthology.orchestra.run.smithi178.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-15T04:52:01.202 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:01.202 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:52:01.417 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T04:52:01.445 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph 2023-11-15T04:52:01.721 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T04:52:02.013 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T04:52:02.014 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-15T04:52:02.014 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-15T04:52:02.014 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-15T04:52:02.014 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-15T04:52:02.014 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-15T04:52:02.314 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:56 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:52:02.772 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-15T04:52:02.772 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:02.773 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2023-11-15T04:52:02.773 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-15T04:52:02.773 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-15T04:52:02.773 INFO:teuthology.orchestra.run.smithi163.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-15T04:52:02.774 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:02.774 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:52:02.994 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph 2023-11-15T04:52:03.849 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:57 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:52:03.989 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:52:03.992 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:52:03.992 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2023-11-15T04:52:03.992 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:52:03.993 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-15T04:52:03.993 INFO:teuthology.orchestra.run.smithi178.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-15T04:52:03.993 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-11-15T04:52:03.993 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-15T04:52:03.993 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-15T04:52:03.994 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-15T04:52:03.994 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-15T04:52:03.994 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-15T04:52:03.994 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-15T04:52:03.994 INFO:teuthology.orchestra.run.smithi178.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-15T04:52:03.995 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-15T04:52:03.995 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-15T04:52:03.995 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-15T04:52:03.995 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-15T04:52:03.995 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-15T04:52:03.996 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-15T04:52:03.996 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-15T04:52:03.996 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-15T04:52:03.996 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-15T04:52:03.996 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-15T04:52:03.997 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-15T04:52:03.997 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-15T04:52:03.997 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-15T04:52:03.997 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-15T04:52:03.997 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-15T04:52:03.998 INFO:teuthology.orchestra.run.smithi178.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-15T04:52:03.998 INFO:teuthology.orchestra.run.smithi178.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-15T04:52:03.998 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-15T04:52:03.998 INFO:teuthology.orchestra.run.smithi178.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-15T04:52:03.998 INFO:teuthology.orchestra.run.smithi178.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-15T04:52:03.999 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-15T04:52:03.999 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-15T04:52:03.999 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-15T04:52:03.999 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-15T04:52:04.000 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:04.000 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-15T04:52:04.000 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:52:04.000 INFO:teuthology.orchestra.run.smithi178.stdout:Install 32 Packages 2023-11-15T04:52:04.000 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:04.001 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 31 M 2023-11-15T04:52:04.001 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 111 M 2023-11-15T04:52:04.002 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-15T04:52:04.621 INFO:teuthology.orchestra.run.smithi178.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 89 kB/s | 6.3 kB 00:00 2023-11-15T04:52:04.663 INFO:teuthology.orchestra.run.smithi178.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2023-11-15T04:52:04.805 INFO:teuthology.orchestra.run.smithi178.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 9.4 MB/s | 2.4 MB 00:00 2023-11-15T04:52:04.830 INFO:teuthology.orchestra.run.smithi178.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2023-11-15T04:52:04.914 INFO:teuthology.orchestra.run.smithi178.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 17 MB/s | 5.0 MB 00:00 2023-11-15T04:52:05.135 INFO:teuthology.orchestra.run.smithi178.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 39 MB/s | 18 MB 00:00 2023-11-15T04:52:05.181 INFO:teuthology.orchestra.run.smithi178.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 448 kB/s | 157 kB 00:00 2023-11-15T04:52:05.256 INFO:teuthology.orchestra.run.smithi178.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 759 kB/s | 57 kB 00:00 2023-11-15T04:52:05.281 INFO:teuthology.orchestra.run.smithi178.stdout:(9/32): python3-markupsafe-0.23-19.el8.x86_64.r 105 kB/s | 39 kB 00:00 2023-11-15T04:52:05.407 INFO:teuthology.orchestra.run.smithi178.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 382 kB/s | 103 kB 00:00 2023-11-15T04:52:05.432 INFO:teuthology.orchestra.run.smithi178.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.6 MB/s | 249 kB 00:00 2023-11-15T04:52:05.458 INFO:teuthology.orchestra.run.smithi178.stdout:(12/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.2 MB/s | 457 kB 00:00 2023-11-15T04:52:05.513 INFO:teuthology.orchestra.run.smithi178.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.3 MB/s | 109 kB 00:00 2023-11-15T04:52:05.547 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:52:05.550 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:52:05.551 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2023-11-15T04:52:05.551 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:52:05.551 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2023-11-15T04:52:05.551 INFO:teuthology.orchestra.run.smithi163.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-15T04:52:05.551 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2023-11-15T04:52:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-15T04:52:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-15T04:52:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-15T04:52:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-15T04:52:05.552 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-15T04:52:05.553 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-15T04:52:05.553 INFO:teuthology.orchestra.run.smithi163.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-15T04:52:05.553 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-15T04:52:05.553 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-15T04:52:05.553 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-15T04:52:05.553 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-15T04:52:05.554 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-15T04:52:05.554 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-15T04:52:05.554 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-15T04:52:05.554 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-15T04:52:05.554 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-15T04:52:05.555 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-15T04:52:05.555 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-15T04:52:05.555 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-15T04:52:05.555 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-15T04:52:05.555 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-15T04:52:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-15T04:52:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-15T04:52:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-15T04:52:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-15T04:52:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-15T04:52:05.556 INFO:teuthology.orchestra.run.smithi163.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-15T04:52:05.557 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-15T04:52:05.557 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-15T04:52:05.557 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-15T04:52:05.557 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-15T04:52:05.558 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:05.558 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2023-11-15T04:52:05.558 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:52:05.558 INFO:teuthology.orchestra.run.smithi163.stdout:Install 32 Packages 2023-11-15T04:52:05.559 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:05.559 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 31 M 2023-11-15T04:52:05.559 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 111 M 2023-11-15T04:52:05.560 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2023-11-15T04:52:05.583 INFO:teuthology.orchestra.run.smithi178.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 3.6 MB/s | 641 kB 00:00 2023-11-15T04:52:05.742 INFO:teuthology.orchestra.run.smithi178.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 153 kB/s | 44 kB 00:00 2023-11-15T04:52:05.792 INFO:teuthology.orchestra.run.smithi178.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 665 kB/s | 185 kB 00:00 2023-11-15T04:52:05.834 INFO:teuthology.orchestra.run.smithi178.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 689 kB/s | 173 kB 00:00 2023-11-15T04:52:05.884 INFO:teuthology.orchestra.run.smithi178.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2023-11-15T04:52:05.901 INFO:teuthology.orchestra.run.smithi178.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 262 kB/s | 18 kB 00:00 2023-11-15T04:52:05.935 INFO:teuthology.orchestra.run.smithi178.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.0 MB/s | 384 kB 00:00 2023-11-15T04:52:05.951 INFO:teuthology.orchestra.run.smithi178.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 725 kB/s | 49 kB 00:00 2023-11-15T04:52:05.977 INFO:teuthology.orchestra.run.smithi178.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 778 kB/s | 59 kB 00:00 2023-11-15T04:52:06.019 INFO:teuthology.orchestra.run.smithi178.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.3 MB/s | 283 kB 00:00 2023-11-15T04:52:06.036 INFO:teuthology.orchestra.run.smithi178.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 185 kB/s | 16 kB 00:00 2023-11-15T04:52:06.053 INFO:teuthology.orchestra.run.smithi178.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 248 kB/s | 19 kB 00:00 2023-11-15T04:52:06.095 INFO:teuthology.orchestra.run.smithi178.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 317 kB/s | 24 kB 00:00 2023-11-15T04:52:06.120 INFO:teuthology.orchestra.run.smithi178.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 348 kB/s | 29 kB 00:00 2023-11-15T04:52:06.145 INFO:teuthology.orchestra.run.smithi178.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 298 kB/s | 27 kB 00:00 2023-11-15T04:52:06.187 INFO:teuthology.orchestra.run.smithi178.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.7 MB/s | 255 kB 00:00 2023-11-15T04:52:06.203 INFO:teuthology.orchestra.run.smithi163.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 110 kB/s | 6.3 kB 00:00 2023-11-15T04:52:06.253 INFO:teuthology.orchestra.run.smithi163.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2023-11-15T04:52:06.254 INFO:teuthology.orchestra.run.smithi178.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 786 kB/s | 85 kB 00:00 2023-11-15T04:52:06.279 INFO:teuthology.orchestra.run.smithi178.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 249 kB/s | 23 kB 00:00 2023-11-15T04:52:06.304 INFO:teuthology.orchestra.run.smithi163.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2023-11-15T04:52:06.304 INFO:teuthology.orchestra.run.smithi178.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.3 MB/s | 251 kB 00:00 2023-11-15T04:52:06.306 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:52:06.306 INFO:teuthology.orchestra.run.smithi178.stdout:Total 13 MB/s | 31 MB 00:02 2023-11-15T04:52:06.362 INFO:teuthology.orchestra.run.smithi163.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 3.7 MB/s | 220 kB 00:00 2023-11-15T04:52:06.390 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-15T04:52:06.429 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-15T04:52:06.430 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-15T04:52:06.446 INFO:teuthology.orchestra.run.smithi163.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2023-11-15T04:52:06.722 INFO:teuthology.orchestra.run.smithi163.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 39 MB/s | 18 MB 00:00 2023-11-15T04:52:06.747 INFO:teuthology.orchestra.run.smithi163.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 129 kB/s | 39 kB 00:00 2023-11-15T04:52:06.770 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-15T04:52:06.772 INFO:teuthology.orchestra.run.smithi163.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 383 kB/s | 157 kB 00:00 2023-11-15T04:52:06.773 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-15T04:52:06.872 INFO:teuthology.orchestra.run.smithi163.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 454 kB/s | 57 kB 00:00 2023-11-15T04:52:06.948 INFO:teuthology.orchestra.run.smithi163.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.5 MB/s | 457 kB 00:00 2023-11-15T04:52:06.998 INFO:teuthology.orchestra.run.smithi163.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.9 MB/s | 249 kB 00:00 2023-11-15T04:52:07.053 INFO:teuthology.orchestra.run.smithi163.stdout:(12/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 312 kB/s | 103 kB 00:00 2023-11-15T04:52:07.091 INFO:teuthology.orchestra.run.smithi163.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.2 MB/s | 109 kB 00:00 2023-11-15T04:52:07.149 INFO:teuthology.orchestra.run.smithi163.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 3.1 MB/s | 641 kB 00:00 2023-11-15T04:52:07.216 INFO:teuthology.orchestra.run.smithi163.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 269 kB/s | 44 kB 00:00 2023-11-15T04:52:07.299 INFO:teuthology.orchestra.run.smithi163.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 912 kB/s | 185 kB 00:00 2023-11-15T04:52:07.350 INFO:teuthology.orchestra.run.smithi163.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 863 kB/s | 173 kB 00:00 2023-11-15T04:52:07.375 INFO:teuthology.orchestra.run.smithi163.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 146 kB/s | 11 kB 00:00 2023-11-15T04:52:07.401 INFO:teuthology.orchestra.run.smithi163.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 349 kB/s | 18 kB 00:00 2023-11-15T04:52:07.426 INFO:teuthology.orchestra.run.smithi163.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 969 kB/s | 49 kB 00:00 2023-11-15T04:52:07.451 INFO:teuthology.orchestra.run.smithi163.stdout:(21/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.6 MB/s | 384 kB 00:00 2023-11-15T04:52:07.477 INFO:teuthology.orchestra.run.smithi163.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 773 kB/s | 59 kB 00:00 2023-11-15T04:52:07.527 INFO:teuthology.orchestra.run.smithi163.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 206 kB/s | 16 kB 00:00 2023-11-15T04:52:07.561 INFO:teuthology.orchestra.run.smithi163.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 224 kB/s | 19 kB 00:00 2023-11-15T04:52:07.586 INFO:teuthology.orchestra.run.smithi163.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 411 kB/s | 24 kB 00:00 2023-11-15T04:52:07.613 INFO:teuthology.orchestra.run.smithi163.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 582 kB/s | 29 kB 00:00 2023-11-15T04:52:07.637 INFO:teuthology.orchestra.run.smithi163.stdout:(27/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2023-11-15T04:52:07.662 INFO:teuthology.orchestra.run.smithi163.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 363 kB/s | 27 kB 00:00 2023-11-15T04:52:07.687 INFO:teuthology.orchestra.run.smithi163.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2023-11-15T04:52:07.713 INFO:teuthology.orchestra.run.smithi163.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.3 MB/s | 251 kB 00:00 2023-11-15T04:52:07.766 INFO:teuthology.orchestra.run.smithi163.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 825 kB/s | 85 kB 00:00 2023-11-15T04:52:07.788 INFO:teuthology.orchestra.run.smithi163.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 229 kB/s | 23 kB 00:00 2023-11-15T04:52:07.792 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:52:07.792 INFO:teuthology.orchestra.run.smithi163.stdout:Total 14 MB/s | 31 MB 00:02 2023-11-15T04:52:07.874 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2023-11-15T04:52:07.914 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2023-11-15T04:52:07.914 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2023-11-15T04:52:07.947 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-15T04:52:08.184 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-15T04:52:08.224 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2023-11-15T04:52:08.227 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2023-11-15T04:52:08.387 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-15T04:52:08.654 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-15T04:52:08.861 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-15T04:52:09.050 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-15T04:52:09.263 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-15T04:52:09.432 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-15T04:52:09.441 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2023-11-15T04:52:09.635 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-15T04:52:09.642 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-15T04:52:09.825 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-15T04:52:09.837 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-15T04:52:10.088 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-15T04:52:10.121 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-15T04:52:10.360 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-15T04:52:10.360 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-15T04:52:10.600 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-15T04:52:10.604 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-15T04:52:10.779 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-15T04:52:10.982 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-15T04:52:10.982 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-15T04:52:11.184 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-15T04:52:11.223 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-15T04:52:11.409 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-15T04:52:11.420 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-15T04:52:11.655 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-15T04:52:11.678 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-15T04:52:11.888 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T04:52:11.939 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-15T04:52:12.204 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-15T04:52:12.210 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-15T04:52:12.409 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-15T04:52:12.600 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-15T04:52:12.623 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-15T04:52:12.866 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-15T04:52:12.906 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-15T04:52:13.106 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-15T04:52:13.128 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-15T04:52:13.350 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-15T04:52:13.402 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-15T04:52:13.596 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T04:52:13.623 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-15T04:52:13.911 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-15T04:52:13.919 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-15T04:52:14.108 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-15T04:52:14.218 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-15T04:52:14.305 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-15T04:52:14.521 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-15T04:52:14.656 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-15T04:52:14.784 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-15T04:52:14.877 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T04:52:15.063 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-15T04:52:15.288 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-15T04:52:15.633 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-15T04:52:15.966 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-15T04:52:16.412 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T04:52:16.429 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-15T04:52:16.627 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T04:52:16.675 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T04:52:17.061 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T04:52:17.335 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T04:52:17.558 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T04:52:17.712 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T04:52:17.729 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T04:52:17.869 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T04:52:18.210 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T04:52:18.301 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T04:52:18.302 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-15T04:52:18.302 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-15T04:52:18.302 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-15T04:52:18.302 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-15T04:52:18.303 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-15T04:52:18.303 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-15T04:52:18.303 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-15T04:52:18.303 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-15T04:52:18.303 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-15T04:52:18.304 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-15T04:52:18.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-15T04:52:18.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-15T04:52:18.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-15T04:52:18.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-15T04:52:18.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-15T04:52:18.306 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-15T04:52:18.306 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T04:52:18.306 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-15T04:52:18.306 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-15T04:52:18.307 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-15T04:52:18.307 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-15T04:52:18.307 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-15T04:52:18.308 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-15T04:52:18.308 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-15T04:52:18.308 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-15T04:52:18.308 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-15T04:52:18.308 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-15T04:52:18.309 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-15T04:52:18.309 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-15T04:52:18.310 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-15T04:52:18.310 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-15T04:52:18.442 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T04:52:18.874 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T04:52:19.142 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T04:52:19.364 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T04:52:19.428 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-15T04:52:19.429 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:19.429 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-15T04:52:19.429 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-15T04:52:19.429 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-15T04:52:19.429 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-15T04:52:19.430 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-15T04:52:19.430 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-15T04:52:19.430 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-15T04:52:19.430 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-15T04:52:19.430 INFO:teuthology.orchestra.run.smithi178.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-15T04:52:19.431 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-15T04:52:19.431 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-15T04:52:19.431 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-15T04:52:19.431 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-15T04:52:19.431 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-15T04:52:19.432 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-15T04:52:19.432 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-15T04:52:19.432 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-15T04:52:19.432 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-15T04:52:19.432 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-15T04:52:19.432 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-15T04:52:19.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend-2.6-1.el8.noarch 2023-11-15T04:52:19.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-15T04:52:19.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-15T04:52:19.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-15T04:52:19.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-15T04:52:19.434 INFO:teuthology.orchestra.run.smithi178.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-15T04:52:19.434 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-15T04:52:19.434 INFO:teuthology.orchestra.run.smithi178.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-15T04:52:19.434 INFO:teuthology.orchestra.run.smithi178.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-15T04:52:19.434 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-15T04:52:19.435 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-15T04:52:19.435 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-15T04:52:19.435 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-15T04:52:19.435 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:19.435 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:52:19.586 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T04:52:19.604 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T04:52:19.684 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-base 2023-11-15T04:52:19.719 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T04:52:20.169 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T04:52:20.170 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-15T04:52:20.170 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-15T04:52:20.170 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-15T04:52:20.170 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-15T04:52:20.171 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-15T04:52:20.171 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-15T04:52:20.171 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-15T04:52:20.171 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-15T04:52:20.171 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-15T04:52:20.172 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-15T04:52:20.172 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-15T04:52:20.172 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-15T04:52:20.172 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-15T04:52:20.172 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-15T04:52:20.173 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-15T04:52:20.173 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-15T04:52:20.173 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T04:52:20.173 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-15T04:52:20.173 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-15T04:52:20.174 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-15T04:52:20.174 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-15T04:52:20.174 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-15T04:52:20.174 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-15T04:52:20.174 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-15T04:52:20.175 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-15T04:52:20.175 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-15T04:52:20.175 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-15T04:52:20.175 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-15T04:52:20.175 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-15T04:52:20.176 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-15T04:52:20.176 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-15T04:52:20.551 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:14 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:52:21.392 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-15T04:52:21.393 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:21.393 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2023-11-15T04:52:21.393 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-15T04:52:21.393 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-15T04:52:21.393 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-15T04:52:21.394 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-15T04:52:21.394 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-15T04:52:21.394 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-15T04:52:21.394 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-15T04:52:21.394 INFO:teuthology.orchestra.run.smithi163.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-15T04:52:21.395 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-15T04:52:21.395 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-15T04:52:21.395 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-15T04:52:21.395 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-15T04:52:21.395 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-15T04:52:21.396 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-15T04:52:21.396 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-15T04:52:21.396 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-15T04:52:21.396 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-15T04:52:21.396 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-15T04:52:21.397 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-15T04:52:21.397 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend-2.6-1.el8.noarch 2023-11-15T04:52:21.397 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-15T04:52:21.397 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-15T04:52:21.398 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-15T04:52:21.398 INFO:teuthology.orchestra.run.smithi163.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-15T04:52:21.398 INFO:teuthology.orchestra.run.smithi163.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-15T04:52:21.398 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-15T04:52:21.398 INFO:teuthology.orchestra.run.smithi163.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-15T04:52:21.399 INFO:teuthology.orchestra.run.smithi163.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-15T04:52:21.399 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-15T04:52:21.399 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-15T04:52:21.399 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-15T04:52:21.399 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-15T04:52:21.400 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:21.400 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:52:22.002 INFO:teuthology.orchestra.run.smithi178.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:52:22.217 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:52:22.219 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2023-11-15T04:52:22.219 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:52:22.315 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install cephadm 2023-11-15T04:52:22.666 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-base 2023-11-15T04:52:23.167 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:17 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:52:23.524 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:17 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:52:24.839 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:52:24.840 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:52:24.840 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2023-11-15T04:52:24.840 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:52:24.841 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-15T04:52:24.841 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-15T04:52:24.841 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:24.841 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-15T04:52:24.841 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:52:24.842 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-15T04:52:24.842 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:24.842 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 71 k 2023-11-15T04:52:24.842 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 290 k 2023-11-15T04:52:24.842 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-15T04:52:24.892 INFO:teuthology.orchestra.run.smithi178.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 71 kB 00:00 2023-11-15T04:52:24.893 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:52:24.893 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.3 MB/s | 71 kB 00:00 2023-11-15T04:52:24.893 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-15T04:52:24.900 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-15T04:52:24.900 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-15T04:52:24.902 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-15T04:52:24.905 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-15T04:52:24.944 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-15T04:52:24.970 INFO:teuthology.orchestra.run.smithi163.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:52:25.184 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:52:25.185 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2023-11-15T04:52:25.185 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:52:25.302 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install cephadm 2023-11-15T04:52:25.473 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:52:25.696 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:52:25.920 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:52:26.154 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:20 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:52:27.182 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:52:27.182 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:27.182 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-15T04:52:27.182 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-15T04:52:27.183 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:27.183 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:52:27.362 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-immutable-object-cache 2023-11-15T04:52:27.821 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:52:27.822 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:52:27.822 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2023-11-15T04:52:27.823 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:52:27.823 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2023-11-15T04:52:27.823 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-15T04:52:27.823 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:27.823 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2023-11-15T04:52:27.824 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:52:27.824 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2023-11-15T04:52:27.824 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:27.824 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 71 k 2023-11-15T04:52:27.824 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 290 k 2023-11-15T04:52:27.825 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2023-11-15T04:52:28.223 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:22 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:52:28.244 INFO:teuthology.orchestra.run.smithi163.stdout:cephadm-16.2.4-0.el8.noarch.rpm 169 kB/s | 71 kB 00:00 2023-11-15T04:52:28.244 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:52:28.245 INFO:teuthology.orchestra.run.smithi163.stdout:Total 169 kB/s | 71 kB 00:00 2023-11-15T04:52:28.245 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2023-11-15T04:52:28.253 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2023-11-15T04:52:28.253 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2023-11-15T04:52:28.255 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2023-11-15T04:52:28.259 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2023-11-15T04:52:28.472 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2023-11-15T04:52:29.047 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:52:29.269 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:52:29.503 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:52:29.860 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:52:29.861 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:52:29.861 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2023-11-15T04:52:29.861 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:52:29.861 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-15T04:52:29.862 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-15T04:52:29.862 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:29.862 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-15T04:52:29.862 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:52:29.862 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-15T04:52:29.863 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:29.863 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 163 k 2023-11-15T04:52:29.863 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 587 k 2023-11-15T04:52:29.864 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-15T04:52:29.921 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 2.7 MB/s | 163 kB 00:00 2023-11-15T04:52:29.922 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:52:29.922 INFO:teuthology.orchestra.run.smithi178.stdout:Total 2.6 MB/s | 163 kB 00:00 2023-11-15T04:52:29.923 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-15T04:52:29.933 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-15T04:52:29.933 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-15T04:52:29.979 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-15T04:52:29.982 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-15T04:52:30.127 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-15T04:52:30.149 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:52:30.150 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:30.150 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2023-11-15T04:52:30.150 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-15T04:52:30.151 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:30.151 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:52:30.309 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:52:30.351 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-immutable-object-cache 2023-11-15T04:52:30.757 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:52:31.210 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:25 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:52:31.456 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:52:31.456 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:31.456 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-15T04:52:31.457 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-15T04:52:31.457 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:31.457 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:52:31.681 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr 2023-11-15T04:52:32.545 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:26 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:52:32.865 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:52:32.866 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:52:32.867 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2023-11-15T04:52:32.867 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:52:32.867 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2023-11-15T04:52:32.867 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-15T04:52:32.867 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:32.868 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2023-11-15T04:52:32.868 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:52:32.868 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2023-11-15T04:52:32.868 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:32.869 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 163 k 2023-11-15T04:52:32.869 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 587 k 2023-11-15T04:52:32.869 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2023-11-15T04:52:32.942 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 2.1 MB/s | 163 kB 00:00 2023-11-15T04:52:32.943 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:52:32.943 INFO:teuthology.orchestra.run.smithi163.stdout:Total 2.1 MB/s | 163 kB 00:00 2023-11-15T04:52:32.943 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2023-11-15T04:52:32.954 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2023-11-15T04:52:32.955 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2023-11-15T04:52:33.002 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2023-11-15T04:52:33.005 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2023-11-15T04:52:33.119 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2023-11-15T04:52:33.325 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:52:33.747 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:52:33.991 INFO:teuthology.orchestra.run.smithi178.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:52:34.206 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:52:34.207 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2023-11-15T04:52:34.207 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:52:34.322 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-dashboard 2023-11-15T04:52:35.173 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:29 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:52:35.212 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:52:35.212 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:35.212 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2023-11-15T04:52:35.213 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-15T04:52:35.213 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:35.213 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:52:35.511 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr 2023-11-15T04:52:36.380 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:30 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:52:36.869 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:52:36.870 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:52:36.870 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2023-11-15T04:52:36.871 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:52:36.871 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-15T04:52:36.871 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-15T04:52:36.871 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-11-15T04:52:36.871 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-15T04:52:36.872 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-15T04:52:36.872 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-15T04:52:36.872 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-15T04:52:36.872 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-15T04:52:36.872 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:36.873 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-15T04:52:36.873 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:52:36.873 INFO:teuthology.orchestra.run.smithi178.stdout:Install 6 Packages 2023-11-15T04:52:36.873 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:36.874 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 3.0 M 2023-11-15T04:52:36.874 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 15 M 2023-11-15T04:52:36.874 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-15T04:52:37.149 INFO:teuthology.orchestra.run.smithi178.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 371 kB/s | 24 kB 00:00 2023-11-15T04:52:37.175 INFO:teuthology.orchestra.run.smithi178.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 99 kB/s | 8.9 kB 00:00 2023-11-15T04:52:37.233 INFO:teuthology.orchestra.run.smithi178.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 18 MB/s | 2.7 MB 00:00 2023-11-15T04:52:37.309 INFO:teuthology.orchestra.run.smithi178.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 268 kB/s | 43 kB 00:00 2023-11-15T04:52:37.450 INFO:teuthology.orchestra.run.smithi178.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 122 kB/s | 34 kB 00:00 2023-11-15T04:52:37.684 INFO:teuthology.orchestra.run.smithi178.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 436 kB/s | 196 kB 00:00 2023-11-15T04:52:37.686 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:52:37.687 INFO:teuthology.orchestra.run.smithi178.stdout:Total 3.6 MB/s | 3.0 MB 00:00 2023-11-15T04:52:37.696 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-15T04:52:37.706 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-15T04:52:37.707 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-15T04:52:37.822 INFO:teuthology.orchestra.run.smithi163.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:52:37.883 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-15T04:52:37.887 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-15T04:52:38.037 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:52:38.038 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2023-11-15T04:52:38.038 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:52:38.153 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-dashboard 2023-11-15T04:52:38.166 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-15T04:52:38.447 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-15T04:52:38.725 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-15T04:52:38.968 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-15T04:52:39.001 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:33 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:52:39.136 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-15T04:52:40.131 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-15T04:52:40.438 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:52:40.653 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:52:40.654 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:52:40.655 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2023-11-15T04:52:40.655 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:52:40.655 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2023-11-15T04:52:40.655 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-15T04:52:40.655 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2023-11-15T04:52:40.656 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-15T04:52:40.656 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-15T04:52:40.656 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-15T04:52:40.656 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-15T04:52:40.656 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-15T04:52:40.657 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:40.657 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2023-11-15T04:52:40.657 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:52:40.657 INFO:teuthology.orchestra.run.smithi163.stdout:Install 6 Packages 2023-11-15T04:52:40.657 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:40.658 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 3.0 M 2023-11-15T04:52:40.658 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 15 M 2023-11-15T04:52:40.659 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2023-11-15T04:52:40.707 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:52:40.707 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-15T04:52:40.708 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-15T04:52:40.708 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-15T04:52:40.708 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-15T04:52:40.708 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-15T04:52:41.073 INFO:teuthology.orchestra.run.smithi163.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 432 kB/s | 24 kB 00:00 2023-11-15T04:52:41.098 INFO:teuthology.orchestra.run.smithi163.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 110 kB/s | 8.9 kB 00:00 2023-11-15T04:52:41.157 INFO:teuthology.orchestra.run.smithi163.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 19 MB/s | 2.7 MB 00:00 2023-11-15T04:52:41.265 INFO:teuthology.orchestra.run.smithi163.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 221 kB/s | 43 kB 00:00 2023-11-15T04:52:41.291 INFO:teuthology.orchestra.run.smithi163.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 175 kB/s | 34 kB 00:00 2023-11-15T04:52:41.324 INFO:teuthology.orchestra.run.smithi163.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.2 MB/s | 196 kB 00:00 2023-11-15T04:52:41.327 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:52:41.327 INFO:teuthology.orchestra.run.smithi163.stdout:Total 4.4 MB/s | 3.0 MB 00:00 2023-11-15T04:52:41.336 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2023-11-15T04:52:41.345 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2023-11-15T04:52:41.346 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2023-11-15T04:52:41.406 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-15T04:52:41.406 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:41.407 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-15T04:52:41.407 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-15T04:52:41.407 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-15T04:52:41.407 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-15T04:52:41.407 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-15T04:52:41.408 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-15T04:52:41.408 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-15T04:52:41.408 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:41.408 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:52:41.496 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2023-11-15T04:52:41.499 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2023-11-15T04:52:41.552 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-15T04:52:41.826 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2023-11-15T04:52:42.071 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-15T04:52:42.333 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-15T04:52:42.408 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:36 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:52:42.567 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-15T04:52:42.743 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-15T04:52:43.775 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-15T04:52:44.058 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:52:44.059 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================================ 2023-11-15T04:52:44.059 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2023-11-15T04:52:44.059 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================================ 2023-11-15T04:52:44.060 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-15T04:52:44.060 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-15T04:52:44.060 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-11-15T04:52:44.060 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-15T04:52:44.060 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-15T04:52:44.061 INFO:teuthology.orchestra.run.smithi178.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-15T04:52:44.061 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-15T04:52:44.061 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-15T04:52:44.061 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-15T04:52:44.061 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-15T04:52:44.062 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:44.062 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-15T04:52:44.062 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================================ 2023-11-15T04:52:44.062 INFO:teuthology.orchestra.run.smithi178.stdout:Install 8 Packages 2023-11-15T04:52:44.062 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:44.063 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 37 M 2023-11-15T04:52:44.063 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 210 M 2023-11-15T04:52:44.063 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-15T04:52:44.070 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:52:44.358 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:52:44.359 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-15T04:52:44.359 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-15T04:52:44.359 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-15T04:52:44.359 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-15T04:52:44.359 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-15T04:52:44.448 INFO:teuthology.orchestra.run.smithi178.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 53 MB/s | 8.6 MB 00:00 2023-11-15T04:52:45.033 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-15T04:52:45.033 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:45.033 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2023-11-15T04:52:45.034 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-15T04:52:45.034 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-15T04:52:45.034 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-15T04:52:45.034 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-15T04:52:45.035 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-15T04:52:45.035 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-15T04:52:45.035 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:45.035 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:52:45.124 INFO:teuthology.orchestra.run.smithi178.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 5.5 MB/s | 3.7 MB 00:00 2023-11-15T04:52:45.171 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-15T04:52:45.190 INFO:teuthology.orchestra.run.smithi178.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 5.2 MB/s | 4.7 MB 00:00 2023-11-15T04:52:45.257 INFO:teuthology.orchestra.run.smithi178.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.0 MB/s | 4.9 MB 00:00 2023-11-15T04:52:45.283 INFO:teuthology.orchestra.run.smithi178.stdout:(5/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.4 MB/s | 225 kB 00:00 2023-11-15T04:52:45.400 INFO:teuthology.orchestra.run.smithi178.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.4 MB/s | 172 kB 00:00 2023-11-15T04:52:45.425 INFO:teuthology.orchestra.run.smithi178.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 3.8 MB/s | 645 kB 00:00 2023-11-15T04:52:45.725 INFO:teuthology.orchestra.run.smithi178.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 26 MB/s | 14 MB 00:00 2023-11-15T04:52:45.726 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:52:45.726 INFO:teuthology.orchestra.run.smithi178.stdout:Total 22 MB/s | 37 MB 00:01 2023-11-15T04:52:45.956 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-15T04:52:45.974 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-15T04:52:45.974 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-15T04:52:46.026 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:40 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:52:46.308 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-15T04:52:46.312 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-15T04:52:46.970 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-15T04:52:47.097 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T04:52:47.220 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T04:52:47.398 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T04:52:47.689 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:52:47.691 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================================ 2023-11-15T04:52:47.692 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2023-11-15T04:52:47.692 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================================ 2023-11-15T04:52:47.692 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2023-11-15T04:52:47.692 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-15T04:52:47.692 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2023-11-15T04:52:47.693 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-15T04:52:47.693 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-15T04:52:47.693 INFO:teuthology.orchestra.run.smithi163.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-15T04:52:47.693 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-15T04:52:47.693 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-15T04:52:47.694 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-15T04:52:47.694 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-15T04:52:47.694 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:47.694 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2023-11-15T04:52:47.694 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================================ 2023-11-15T04:52:47.695 INFO:teuthology.orchestra.run.smithi163.stdout:Install 8 Packages 2023-11-15T04:52:47.695 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:47.695 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 37 M 2023-11-15T04:52:47.695 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 210 M 2023-11-15T04:52:47.696 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2023-11-15T04:52:47.996 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T04:52:48.099 INFO:teuthology.orchestra.run.smithi163.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 48 MB/s | 8.6 MB 00:00 2023-11-15T04:52:48.197 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:52:48.674 INFO:teuthology.orchestra.run.smithi163.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 6.3 MB/s | 4.7 MB 00:00 2023-11-15T04:52:48.776 INFO:teuthology.orchestra.run.smithi163.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.7 MB/s | 4.9 MB 00:00 2023-11-15T04:52:48.812 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:52:48.833 INFO:teuthology.orchestra.run.smithi163.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 5.1 MB/s | 3.7 MB 00:00 2023-11-15T04:52:48.860 INFO:teuthology.orchestra.run.smithi163.stdout:(5/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.6 MB/s | 225 kB 00:00 2023-11-15T04:52:48.967 INFO:teuthology.orchestra.run.smithi163.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.6 MB/s | 172 kB 00:00 2023-11-15T04:52:49.001 INFO:teuthology.orchestra.run.smithi163.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 3.8 MB/s | 645 kB 00:00 2023-11-15T04:52:49.262 INFO:teuthology.orchestra.run.smithi163.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 28 MB/s | 14 MB 00:00 2023-11-15T04:52:49.262 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:52:49.262 INFO:teuthology.orchestra.run.smithi163.stdout:Total 23 MB/s | 37 MB 00:01 2023-11-15T04:52:49.489 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2023-11-15T04:52:49.508 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2023-11-15T04:52:49.508 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2023-11-15T04:52:49.789 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T04:52:49.841 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2023-11-15T04:52:49.844 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2023-11-15T04:52:50.471 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T04:52:50.519 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2023-11-15T04:52:50.841 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T04:52:51.125 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T04:52:51.427 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T04:52:51.742 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T04:52:51.930 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T04:52:52.559 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T04:52:52.788 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:52:52.890 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T04:52:53.391 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:52:53.563 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T04:52:54.194 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-15T04:52:54.268 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T04:52:54.416 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T04:52:54.691 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T04:52:54.737 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T04:52:54.737 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-15T04:52:54.738 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-15T04:52:54.738 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:52:54.738 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-15T04:52:54.738 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T04:52:54.738 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-15T04:52:54.739 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-15T04:52:54.954 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T04:52:55.496 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-15T04:52:55.497 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:55.497 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-15T04:52:55.497 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-15T04:52:55.497 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-15T04:52:55.498 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-15T04:52:55.498 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-15T04:52:55.498 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-15T04:52:55.498 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-15T04:52:55.498 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-15T04:52:55.499 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-15T04:52:55.499 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:55.499 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:52:55.607 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-rook 2023-11-15T04:52:56.465 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:50 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:52:56.717 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T04:52:58.119 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-15T04:52:58.128 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:52:58.130 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:52:58.130 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2023-11-15T04:52:58.131 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:52:58.131 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-15T04:52:58.131 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-15T04:52:58.131 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-11-15T04:52:58.131 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-15T04:52:58.132 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-15T04:52:58.132 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-15T04:52:58.132 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-15T04:52:58.132 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-15T04:52:58.132 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-15T04:52:58.133 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-15T04:52:58.133 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-15T04:52:58.133 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-15T04:52:58.133 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-15T04:52:58.133 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-15T04:52:58.134 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-15T04:52:58.134 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:58.135 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-15T04:52:58.135 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:52:58.135 INFO:teuthology.orchestra.run.smithi178.stdout:Install 13 Packages 2023-11-15T04:52:58.135 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:52:58.136 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 1.9 M 2023-11-15T04:52:58.136 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 24 M 2023-11-15T04:52:58.136 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-15T04:52:58.354 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T04:52:58.457 INFO:teuthology.orchestra.run.smithi178.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 540 kB/s | 27 kB 00:00 2023-11-15T04:52:58.632 INFO:teuthology.orchestra.run.smithi178.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 89 kB/s | 20 kB 00:00 2023-11-15T04:52:58.649 INFO:teuthology.orchestra.run.smithi178.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 110 kB/s | 27 kB 00:00 2023-11-15T04:52:58.651 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T04:52:58.651 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-15T04:52:58.651 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-15T04:52:58.651 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:52:58.652 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-15T04:52:58.652 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T04:52:58.652 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-15T04:52:58.652 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-15T04:52:58.758 INFO:teuthology.orchestra.run.smithi178.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.4 MB/s | 155 kB 00:00 2023-11-15T04:52:58.791 INFO:teuthology.orchestra.run.smithi178.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.3 MB/s | 43 kB 00:00 2023-11-15T04:52:58.825 INFO:teuthology.orchestra.run.smithi178.stdout:(6/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 343 kB/s | 126 kB 00:00 2023-11-15T04:52:58.850 INFO:teuthology.orchestra.run.smithi178.stdout:(7/13): python3-pyasn1-modules-0.3.7-6.el8.noar 505 kB/s | 110 kB 00:00 2023-11-15T04:52:59.142 INFO:teuthology.orchestra.run.smithi178.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 94 kB/s | 33 kB 00:00 2023-11-15T04:52:59.398 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-15T04:52:59.398 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:59.399 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2023-11-15T04:52:59.399 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-15T04:52:59.399 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-15T04:52:59.399 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-15T04:52:59.399 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-15T04:52:59.400 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-15T04:52:59.400 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-15T04:52:59.400 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-15T04:52:59.400 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-15T04:52:59.400 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:52:59.401 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:52:59.547 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-rook 2023-11-15T04:52:59.943 INFO:teuthology.orchestra.run.smithi178.stdout:(9/13): python3-kubernetes-11.0.0-6.el8.noarch. 1.5 MB/s | 1.2 MB 00:00 2023-11-15T04:53:00.285 INFO:teuthology.orchestra.run.smithi178.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 188 kB/s | 64 kB 00:00 2023-11-15T04:53:00.343 INFO:teuthology.orchestra.run.smithi178.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2023-11-15T04:53:00.377 INFO:teuthology.orchestra.run.smithi178.stdout:(12/13): python3-certifi-2018.10.15-7.el8.noarc 10 kB/s | 16 kB 00:01 2023-11-15T04:53:00.410 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:54 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:00.535 INFO:teuthology.orchestra.run.smithi178.stdout:(13/13): python3-google-auth-1.1.1-10.el8.noarc 48 kB/s | 81 kB 00:01 2023-11-15T04:53:00.536 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:53:00.536 INFO:teuthology.orchestra.run.smithi178.stdout:Total 814 kB/s | 1.9 MB 00:02 2023-11-15T04:53:00.594 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-15T04:53:00.610 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-15T04:53:00.611 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-15T04:53:00.697 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-15T04:53:00.700 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-15T04:53:01.214 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-15T04:53:01.455 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-15T04:53:01.675 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-15T04:53:01.922 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-15T04:53:02.063 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:53:02.064 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:53:02.065 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2023-11-15T04:53:02.065 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:53:02.065 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2023-11-15T04:53:02.065 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-15T04:53:02.065 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2023-11-15T04:53:02.066 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-15T04:53:02.066 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-15T04:53:02.066 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-15T04:53:02.066 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-15T04:53:02.066 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-15T04:53:02.067 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-15T04:53:02.067 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-15T04:53:02.067 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-15T04:53:02.067 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-15T04:53:02.067 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-15T04:53:02.068 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-15T04:53:02.068 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-15T04:53:02.068 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:02.068 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2023-11-15T04:53:02.068 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:53:02.069 INFO:teuthology.orchestra.run.smithi163.stdout:Install 13 Packages 2023-11-15T04:53:02.069 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:02.070 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 1.9 M 2023-11-15T04:53:02.071 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 24 M 2023-11-15T04:53:02.071 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2023-11-15T04:53:02.151 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-15T04:53:02.403 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-15T04:53:02.614 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-15T04:53:02.939 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-15T04:53:02.976 INFO:teuthology.orchestra.run.smithi163.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 411 kB/s | 27 kB 00:00 2023-11-15T04:53:03.145 INFO:teuthology.orchestra.run.smithi163.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 85 kB/s | 20 kB 00:00 2023-11-15T04:53:03.209 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-15T04:53:03.221 INFO:teuthology.orchestra.run.smithi163.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 86 kB/s | 27 kB 00:00 2023-11-15T04:53:03.338 INFO:teuthology.orchestra.run.smithi163.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.3 MB/s | 155 kB 00:00 2023-11-15T04:53:03.356 INFO:teuthology.orchestra.run.smithi163.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 335 kB/s | 126 kB 00:00 2023-11-15T04:53:03.379 INFO:teuthology.orchestra.run.smithi163.stdout:(6/13): python3-pyasn1-modules-0.3.7-6.el8.noar 469 kB/s | 110 kB 00:00 2023-11-15T04:53:03.405 INFO:teuthology.orchestra.run.smithi163.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 633 kB/s | 43 kB 00:00 2023-11-15T04:53:03.480 INFO:teuthology.orchestra.run.smithi163.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 163 kB/s | 16 kB 00:00 2023-11-15T04:53:03.506 INFO:teuthology.orchestra.run.smithi163.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 218 kB/s | 33 kB 00:00 2023-11-15T04:53:03.531 INFO:teuthology.orchestra.run.smithi163.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 648 kB/s | 81 kB 00:00 2023-11-15T04:53:03.564 INFO:teuthology.orchestra.run.smithi163.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 1.1 MB/s | 64 kB 00:00 2023-11-15T04:53:03.589 INFO:teuthology.orchestra.run.smithi163.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2023-11-15T04:53:03.677 INFO:teuthology.orchestra.run.smithi163.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.9 MB/s | 1.2 MB 00:00 2023-11-15T04:53:03.678 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:53:03.678 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.2 MB/s | 1.9 MB 00:01 2023-11-15T04:53:03.737 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2023-11-15T04:53:03.752 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2023-11-15T04:53:03.753 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2023-11-15T04:53:03.841 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2023-11-15T04:53:03.844 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2023-11-15T04:53:04.240 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-15T04:53:04.384 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2023-11-15T04:53:04.518 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-15T04:53:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-15T04:53:04.730 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-15T04:53:04.836 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-15T04:53:04.969 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-15T04:53:05.113 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-15T04:53:05.144 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T04:53:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-15T04:53:05.430 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T04:53:05.431 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-15T04:53:05.431 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-15T04:53:05.431 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-15T04:53:05.432 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-15T04:53:05.432 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-15T04:53:05.433 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-15T04:53:05.433 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-15T04:53:05.433 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-15T04:53:05.434 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-15T04:53:05.434 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-15T04:53:05.434 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-15T04:53:05.435 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-15T04:53:05.634 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-15T04:53:05.899 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-15T04:53:06.191 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-15T04:53:06.192 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:06.192 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-15T04:53:06.192 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-15T04:53:06.192 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-15T04:53:06.193 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-15T04:53:06.193 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-15T04:53:06.193 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-15T04:53:06.193 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-15T04:53:06.193 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-15T04:53:06.194 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-15T04:53:06.194 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-15T04:53:06.194 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-15T04:53:06.194 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-15T04:53:06.194 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-15T04:53:06.195 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-15T04:53:06.195 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:06.195 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:53:06.215 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-15T04:53:06.305 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-cephadm 2023-11-15T04:53:06.482 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-15T04:53:07.178 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:01 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:07.556 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-15T04:53:07.842 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-15T04:53:08.075 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-15T04:53:08.310 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-15T04:53:08.502 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T04:53:08.818 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T04:53:08.818 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-15T04:53:08.818 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-15T04:53:08.819 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-15T04:53:08.819 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-15T04:53:08.819 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-15T04:53:08.819 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-15T04:53:08.819 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-15T04:53:08.820 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-15T04:53:08.820 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-15T04:53:08.820 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-15T04:53:08.820 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-15T04:53:08.820 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-15T04:53:08.847 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:53:08.848 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:53:08.848 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2023-11-15T04:53:08.848 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:53:08.849 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-15T04:53:08.849 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-15T04:53:08.849 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-11-15T04:53:08.850 INFO:teuthology.orchestra.run.smithi178.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-15T04:53:08.850 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-15T04:53:08.850 INFO:teuthology.orchestra.run.smithi178.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-15T04:53:08.851 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-15T04:53:08.851 INFO:teuthology.orchestra.run.smithi178.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-15T04:53:08.851 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:08.851 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-15T04:53:08.852 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:53:08.852 INFO:teuthology.orchestra.run.smithi178.stdout:Install 6 Packages 2023-11-15T04:53:08.852 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:08.853 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 5.8 M 2023-11-15T04:53:08.853 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 24 M 2023-11-15T04:53:08.853 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-15T04:53:09.663 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-15T04:53:09.663 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:09.663 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2023-11-15T04:53:09.663 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-15T04:53:09.664 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-15T04:53:09.664 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-15T04:53:09.664 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-15T04:53:09.664 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-15T04:53:09.664 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-15T04:53:09.665 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-15T04:53:09.665 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-15T04:53:09.665 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-15T04:53:09.665 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-15T04:53:09.665 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-15T04:53:09.665 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-15T04:53:09.666 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-15T04:53:09.666 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:09.666 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:53:09.784 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-cephadm 2023-11-15T04:53:10.654 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:04 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:12.322 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:53:12.323 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:53:12.324 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2023-11-15T04:53:12.324 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:53:12.324 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2023-11-15T04:53:12.324 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-15T04:53:12.324 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2023-11-15T04:53:12.325 INFO:teuthology.orchestra.run.smithi163.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-15T04:53:12.325 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-15T04:53:12.325 INFO:teuthology.orchestra.run.smithi163.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-15T04:53:12.325 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-15T04:53:12.325 INFO:teuthology.orchestra.run.smithi163.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-15T04:53:12.326 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:12.326 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2023-11-15T04:53:12.326 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:53:12.326 INFO:teuthology.orchestra.run.smithi163.stdout:Install 6 Packages 2023-11-15T04:53:12.327 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:12.327 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 5.8 M 2023-11-15T04:53:12.327 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 24 M 2023-11-15T04:53:12.328 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2023-11-15T04:53:12.386 INFO:teuthology.orchestra.run.smithi178.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.2 MB/s | 89 kB 00:00 2023-11-15T04:53:12.544 INFO:teuthology.orchestra.run.smithi178.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 134 kB/s | 21 kB 00:00 2023-11-15T04:53:12.724 INFO:teuthology.orchestra.run.smithi163.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.9 MB/s | 89 kB 00:00 2023-11-15T04:53:12.728 INFO:teuthology.orchestra.run.smithi178.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.9 MB/s | 355 kB 00:00 2023-11-15T04:53:12.753 INFO:teuthology.orchestra.run.smithi178.stdout:(4/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.3 MB/s | 538 kB 00:00 2023-11-15T04:53:12.795 INFO:teuthology.orchestra.run.smithi178.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 732 kB/s | 49 kB 00:00 2023-11-15T04:53:12.962 INFO:teuthology.orchestra.run.smithi178.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 7.8 MB/s | 4.8 MB 00:00 2023-11-15T04:53:12.963 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:53:12.963 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.4 MB/s | 5.8 MB 00:04 2023-11-15T04:53:13.000 INFO:teuthology.orchestra.run.smithi163.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 77 kB/s | 21 kB 00:00 2023-11-15T04:53:13.017 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-15T04:53:13.028 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-15T04:53:13.028 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-15T04:53:13.050 INFO:teuthology.orchestra.run.smithi163.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.4 MB/s | 538 kB 00:00 2023-11-15T04:53:13.128 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-15T04:53:13.131 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-15T04:53:13.251 INFO:teuthology.orchestra.run.smithi163.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 8.4 MB/s | 4.8 MB 00:00 2023-11-15T04:53:13.370 INFO:teuthology.orchestra.run.smithi163.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 962 kB/s | 355 kB 00:00 2023-11-15T04:53:13.401 INFO:teuthology.orchestra.run.smithi163.stdout:(6/6): python3-remoto-1.2.1-1.el8.noarch.rpm 139 kB/s | 49 kB 00:00 2023-11-15T04:53:13.402 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:53:13.402 INFO:teuthology.orchestra.run.smithi163.stdout:Total 5.4 MB/s | 5.8 MB 00:01 2023-11-15T04:53:13.439 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-15T04:53:13.457 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2023-11-15T04:53:13.467 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2023-11-15T04:53:13.468 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2023-11-15T04:53:13.584 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2023-11-15T04:53:13.587 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2023-11-15T04:53:13.708 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-15T04:53:13.871 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2023-11-15T04:53:14.033 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-15T04:53:14.159 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-15T04:53:14.464 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-15T04:53:14.960 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-15T04:53:15.407 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-15T04:53:15.416 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-15T04:53:15.675 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-15T04:53:15.804 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-15T04:53:15.900 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:53:16.071 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-15T04:53:16.202 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:53:16.202 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-15T04:53:16.202 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-15T04:53:16.202 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-15T04:53:16.203 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-15T04:53:16.203 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-15T04:53:16.301 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:53:16.594 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:53:16.594 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-15T04:53:16.594 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-15T04:53:16.595 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-15T04:53:16.595 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-15T04:53:16.595 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-15T04:53:16.911 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-15T04:53:16.911 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:16.911 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-15T04:53:16.912 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-15T04:53:16.912 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-15T04:53:16.912 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-15T04:53:16.912 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:16.912 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:53:17.045 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-fuse 2023-11-15T04:53:17.295 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-15T04:53:17.296 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:17.296 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2023-11-15T04:53:17.296 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-15T04:53:17.296 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-15T04:53:17.297 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-15T04:53:17.297 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:17.297 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:53:17.564 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-fuse 2023-11-15T04:53:17.905 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:11 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:18.422 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:12 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:19.570 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:53:19.571 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:53:19.572 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2023-11-15T04:53:19.572 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:53:19.572 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-15T04:53:19.572 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-15T04:53:19.572 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-11-15T04:53:19.573 INFO:teuthology.orchestra.run.smithi178.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-15T04:53:19.573 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:19.573 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-15T04:53:19.573 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:53:19.573 INFO:teuthology.orchestra.run.smithi178.stdout:Install 2 Packages 2023-11-15T04:53:19.574 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:19.574 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 968 k 2023-11-15T04:53:19.574 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 3.2 M 2023-11-15T04:53:19.574 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-15T04:53:19.854 INFO:teuthology.orchestra.run.smithi178.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 918 kB/s | 83 kB 00:00 2023-11-15T04:53:20.106 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:53:20.107 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:53:20.107 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2023-11-15T04:53:20.107 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:53:20.107 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2023-11-15T04:53:20.107 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-15T04:53:20.108 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2023-11-15T04:53:20.108 INFO:teuthology.orchestra.run.smithi163.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-15T04:53:20.108 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:20.108 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2023-11-15T04:53:20.108 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:53:20.109 INFO:teuthology.orchestra.run.smithi163.stdout:Install 2 Packages 2023-11-15T04:53:20.109 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:20.110 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 968 k 2023-11-15T04:53:20.110 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 3.2 M 2023-11-15T04:53:20.110 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2023-11-15T04:53:20.356 INFO:teuthology.orchestra.run.smithi163.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2023-11-15T04:53:20.467 INFO:teuthology.orchestra.run.smithi163.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 498 kB/s | 83 kB 00:00 2023-11-15T04:53:20.468 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:53:20.468 INFO:teuthology.orchestra.run.smithi163.stdout:Total 2.6 MB/s | 968 kB 00:00 2023-11-15T04:53:20.478 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2023-11-15T04:53:20.499 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2023-11-15T04:53:20.500 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2023-11-15T04:53:20.585 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2023-11-15T04:53:20.588 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2023-11-15T04:53:20.759 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2023-11-15T04:53:21.067 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-15T04:53:21.275 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T04:53:21.398 INFO:teuthology.orchestra.run.smithi178.stdout:(2/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 542 kB/s | 885 kB 00:01 2023-11-15T04:53:21.398 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:53:21.399 INFO:teuthology.orchestra.run.smithi178.stdout:Total 530 kB/s | 968 kB 00:01 2023-11-15T04:53:21.410 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-15T04:53:21.431 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-15T04:53:21.432 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-15T04:53:21.524 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-15T04:53:21.527 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-15T04:53:21.653 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T04:53:21.654 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-15T04:53:21.693 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-15T04:53:22.010 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-15T04:53:22.241 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T04:53:22.321 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-15T04:53:22.321 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:22.321 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2023-11-15T04:53:22.322 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-15T04:53:22.322 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:22.322 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:53:22.632 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T04:53:22.632 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-15T04:53:22.951 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librados-devel 2023-11-15T04:53:23.284 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-15T04:53:23.284 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:23.285 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-15T04:53:23.285 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-15T04:53:23.285 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:23.285 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:53:23.514 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados-devel 2023-11-15T04:53:23.819 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:17 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:24.389 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:18 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:25.459 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:53:25.460 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:53:25.461 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2023-11-15T04:53:25.461 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:53:25.461 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2023-11-15T04:53:25.461 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-15T04:53:25.461 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:25.462 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2023-11-15T04:53:25.462 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:53:25.462 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2023-11-15T04:53:25.462 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:25.462 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 131 k 2023-11-15T04:53:25.463 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 439 k 2023-11-15T04:53:25.463 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2023-11-15T04:53:25.503 INFO:teuthology.orchestra.run.smithi163.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 3.1 MB/s | 131 kB 00:00 2023-11-15T04:53:25.504 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:53:25.504 INFO:teuthology.orchestra.run.smithi163.stdout:Total 3.0 MB/s | 131 kB 00:00 2023-11-15T04:53:25.505 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2023-11-15T04:53:25.509 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2023-11-15T04:53:25.510 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2023-11-15T04:53:25.559 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2023-11-15T04:53:25.562 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2023-11-15T04:53:25.668 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2023-11-15T04:53:25.932 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:26.066 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:53:26.067 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:53:26.067 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2023-11-15T04:53:26.067 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:53:26.067 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-15T04:53:26.067 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-15T04:53:26.068 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:26.068 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-15T04:53:26.068 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:53:26.068 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-15T04:53:26.068 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:26.069 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 131 k 2023-11-15T04:53:26.069 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 439 k 2023-11-15T04:53:26.070 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-15T04:53:26.143 INFO:teuthology.orchestra.run.smithi178.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 1.7 MB/s | 131 kB 00:00 2023-11-15T04:53:26.143 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:53:26.144 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.7 MB/s | 131 kB 00:00 2023-11-15T04:53:26.144 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-15T04:53:26.149 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-15T04:53:26.149 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-15T04:53:26.199 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-15T04:53:26.202 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-15T04:53:26.228 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:26.325 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-15T04:53:26.901 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:26.901 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:26.901 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2023-11-15T04:53:26.902 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-15T04:53:26.902 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:26.902 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:53:27.114 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install libcephfs2 2023-11-15T04:53:27.574 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:27.989 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:21 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:28.046 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:28.715 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:28.715 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:28.716 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-15T04:53:28.716 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-15T04:53:28.716 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:28.716 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:53:28.934 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install libcephfs2 2023-11-15T04:53:29.442 INFO:teuthology.orchestra.run.smithi163.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:53:29.656 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:53:29.657 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2023-11-15T04:53:29.657 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:53:29.808 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install libcephfs-devel 2023-11-15T04:53:29.808 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:23 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:30.681 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:24 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:31.263 INFO:teuthology.orchestra.run.smithi178.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:53:31.479 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:53:31.480 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2023-11-15T04:53:31.480 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:53:31.574 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install libcephfs-devel 2023-11-15T04:53:32.358 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:53:32.358 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:53:32.359 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2023-11-15T04:53:32.359 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:53:32.359 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2023-11-15T04:53:32.359 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-15T04:53:32.359 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:32.360 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2023-11-15T04:53:32.360 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:53:32.360 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2023-11-15T04:53:32.360 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:32.361 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 24 k 2023-11-15T04:53:32.361 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 94 k 2023-11-15T04:53:32.361 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2023-11-15T04:53:32.420 INFO:teuthology.orchestra.run.smithi163.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 404 kB/s | 24 kB 00:00 2023-11-15T04:53:32.421 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:53:32.421 INFO:teuthology.orchestra.run.smithi163.stdout:Total 396 kB/s | 24 kB 00:00 2023-11-15T04:53:32.421 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2023-11-15T04:53:32.424 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2023-11-15T04:53:32.424 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2023-11-15T04:53:32.426 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2023-11-15T04:53:32.430 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2023-11-15T04:53:32.438 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:26 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:32.476 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2023-11-15T04:53:32.714 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:32.977 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:33.655 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:33.655 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:33.655 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2023-11-15T04:53:33.655 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-15T04:53:33.656 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:33.656 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:53:33.777 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librados2 2023-11-15T04:53:34.129 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:53:34.130 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:53:34.131 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2023-11-15T04:53:34.131 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:53:34.131 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-15T04:53:34.131 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-15T04:53:34.132 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:34.132 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-15T04:53:34.132 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:53:34.132 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-15T04:53:34.133 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:34.133 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 24 k 2023-11-15T04:53:34.133 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 94 k 2023-11-15T04:53:34.134 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-15T04:53:34.173 INFO:teuthology.orchestra.run.smithi178.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 582 kB/s | 24 kB 00:00 2023-11-15T04:53:34.174 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:53:34.174 INFO:teuthology.orchestra.run.smithi178.stdout:Total 566 kB/s | 24 kB 00:00 2023-11-15T04:53:34.174 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-15T04:53:34.177 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-15T04:53:34.177 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-15T04:53:34.179 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-15T04:53:34.182 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-15T04:53:34.235 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-15T04:53:34.454 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:34.636 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:28 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:34.735 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:35.387 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:35.388 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:35.388 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-15T04:53:35.388 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-15T04:53:35.388 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:35.388 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:53:35.512 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados2 2023-11-15T04:53:36.065 INFO:teuthology.orchestra.run.smithi163.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:53:36.282 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:53:36.283 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2023-11-15T04:53:36.283 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:53:36.373 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:30 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:36.403 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librbd1 2023-11-15T04:53:37.266 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:31 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:37.825 INFO:teuthology.orchestra.run.smithi178.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:53:38.040 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:53:38.041 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2023-11-15T04:53:38.041 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:53:38.139 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librbd1 2023-11-15T04:53:38.722 INFO:teuthology.orchestra.run.smithi163.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:53:38.937 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:53:38.938 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2023-11-15T04:53:38.939 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:53:39.005 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:33 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:39.040 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rados 2023-11-15T04:53:39.903 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:33 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:40.440 INFO:teuthology.orchestra.run.smithi178.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:53:40.656 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:53:40.657 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2023-11-15T04:53:40.658 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:53:40.747 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rados 2023-11-15T04:53:41.322 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:53:41.537 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:53:41.538 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2023-11-15T04:53:41.539 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:53:41.611 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:35 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:41.639 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rgw 2023-11-15T04:53:42.499 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:36 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:43.038 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:53:43.253 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:53:43.254 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2023-11-15T04:53:43.255 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:53:43.384 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rgw 2023-11-15T04:53:43.948 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:53:44.165 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:53:44.166 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2023-11-15T04:53:44.166 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:53:44.243 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:38 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:44.270 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-cephfs 2023-11-15T04:53:45.142 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:39 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:45.668 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:53:45.884 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:53:45.885 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2023-11-15T04:53:45.885 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:53:45.988 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-cephfs 2023-11-15T04:53:46.613 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:53:46.827 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:53:46.828 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2023-11-15T04:53:46.829 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:53:46.852 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:40 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:46.929 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rbd 2023-11-15T04:53:47.788 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:41 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:48.310 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:53:48.524 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:53:48.525 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2023-11-15T04:53:48.525 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:53:48.614 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rbd 2023-11-15T04:53:49.238 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:53:49.458 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:53:49.459 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2023-11-15T04:53:49.459 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:53:49.478 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:43 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:49.559 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-fuse 2023-11-15T04:53:50.422 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:44 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:50.919 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:53:51.133 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:53:51.134 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2023-11-15T04:53:51.134 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:53:51.229 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-fuse 2023-11-15T04:53:52.079 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:53:52.079 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:53:52.080 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2023-11-15T04:53:52.080 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:53:52.080 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2023-11-15T04:53:52.080 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-15T04:53:52.081 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:52.081 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2023-11-15T04:53:52.081 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:53:52.081 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2023-11-15T04:53:52.081 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:52.082 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 89 k 2023-11-15T04:53:52.082 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 298 k 2023-11-15T04:53:52.082 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2023-11-15T04:53:52.104 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:46 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:52.119 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 89 kB 00:00 2023-11-15T04:53:52.120 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:53:52.120 INFO:teuthology.orchestra.run.smithi163.stdout:Total 2.2 MB/s | 89 kB 00:00 2023-11-15T04:53:52.121 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2023-11-15T04:53:52.130 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2023-11-15T04:53:52.130 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2023-11-15T04:53:52.179 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2023-11-15T04:53:52.182 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2023-11-15T04:53:52.324 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2023-11-15T04:53:52.527 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:52.806 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:53.464 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:53.464 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:53.464 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2023-11-15T04:53:53.465 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-15T04:53:53.465 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:53.465 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:53:53.698 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-mirror 2023-11-15T04:53:53.780 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:53:53.781 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:53:53.781 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2023-11-15T04:53:53.781 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:53:53.781 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-15T04:53:53.782 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-15T04:53:53.782 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:53.782 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-15T04:53:53.782 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:53:53.782 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-15T04:53:53.783 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:53.783 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 89 k 2023-11-15T04:53:53.783 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 298 k 2023-11-15T04:53:53.783 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-15T04:53:53.871 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 997 kB/s | 89 kB 00:00 2023-11-15T04:53:53.872 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:53:53.872 INFO:teuthology.orchestra.run.smithi178.stdout:Total 984 kB/s | 89 kB 00:00 2023-11-15T04:53:53.873 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-15T04:53:53.881 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-15T04:53:53.881 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-15T04:53:53.931 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-15T04:53:53.934 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-15T04:53:54.052 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-15T04:53:54.278 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:54.564 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:48 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:54.566 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:55.451 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:55.452 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:55.452 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-15T04:53:55.452 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-15T04:53:55.452 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:55.453 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:53:55.640 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-mirror 2023-11-15T04:53:56.240 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:53:56.241 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:53:56.241 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2023-11-15T04:53:56.242 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:53:56.242 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2023-11-15T04:53:56.242 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T04:53:56.242 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:56.242 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2023-11-15T04:53:56.243 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:53:56.243 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2023-11-15T04:53:56.243 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:56.243 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 3.7 M 2023-11-15T04:53:56.243 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 18 M 2023-11-15T04:53:56.244 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2023-11-15T04:53:56.323 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 46 MB/s | 3.7 MB 00:00 2023-11-15T04:53:56.324 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:53:56.324 INFO:teuthology.orchestra.run.smithi163.stdout:Total 45 MB/s | 3.7 MB 00:00 2023-11-15T04:53:56.325 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2023-11-15T04:53:56.336 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2023-11-15T04:53:56.337 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2023-11-15T04:53:56.399 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2023-11-15T04:53:56.402 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2023-11-15T04:53:56.512 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:50 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:56.895 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2023-11-15T04:53:57.103 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:57.534 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:58.183 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:53:58.184 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:53:58.185 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2023-11-15T04:53:58.185 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:53:58.185 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-15T04:53:58.185 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T04:53:58.185 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:58.186 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-15T04:53:58.186 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:53:58.186 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-15T04:53:58.186 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:53:58.186 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 3.7 M 2023-11-15T04:53:58.187 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 18 M 2023-11-15T04:53:58.187 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-15T04:53:58.189 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:58.190 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:58.190 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2023-11-15T04:53:58.190 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-15T04:53:58.190 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:53:58.190 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:53:58.272 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 43 MB/s | 3.7 MB 00:00 2023-11-15T04:53:58.272 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:53:58.273 INFO:teuthology.orchestra.run.smithi178.stdout:Total 42 MB/s | 3.7 MB 00:00 2023-11-15T04:53:58.273 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-15T04:53:58.285 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-15T04:53:58.285 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-15T04:53:58.349 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-15T04:53:58.352 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-15T04:53:58.440 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-nbd 2023-11-15T04:53:58.816 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-15T04:53:59.076 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:53:59.311 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:53 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:53:59.479 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:54:00.152 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:54:00.153 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:54:00.153 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-15T04:54:00.153 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-15T04:54:00.153 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:54:00.153 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:54:00.428 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-nbd 2023-11-15T04:54:00.979 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-11-15T04:54:00.980 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:54:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2023-11-15T04:54:00.981 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:54:00.981 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2023-11-15T04:54:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-15T04:54:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:54:00.981 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2023-11-15T04:54:00.982 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2023-11-15T04:54:00.982 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2023-11-15T04:54:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:54:00.982 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 173 k 2023-11-15T04:54:00.982 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 649 k 2023-11-15T04:54:00.983 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2023-11-15T04:54:01.029 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.6 MB/s | 173 kB 00:00 2023-11-15T04:54:01.029 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:54:01.030 INFO:teuthology.orchestra.run.smithi163.stdout:Total 3.5 MB/s | 173 kB 00:00 2023-11-15T04:54:01.030 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2023-11-15T04:54:01.040 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2023-11-15T04:54:01.040 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2023-11-15T04:54:01.090 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2023-11-15T04:54:01.092 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2023-11-15T04:54:01.233 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2023-11-15T04:54:01.297 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:55 ago on Wed 15 Nov 2023 04:51:06 AM UTC. 2023-11-15T04:54:01.467 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:54:01.762 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:54:02.469 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:54:02.469 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:54:02.470 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2023-11-15T04:54:02.470 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-15T04:54:02.470 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-11-15T04:54:02.470 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-11-15T04:54:02.726 DEBUG:teuthology.parallel:result is None 2023-11-15T04:54:02.953 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-11-15T04:54:02.954 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:54:02.954 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2023-11-15T04:54:02.955 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:54:02.955 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-11-15T04:54:02.955 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-15T04:54:02.955 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:54:02.956 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-11-15T04:54:02.956 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-11-15T04:54:02.956 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2023-11-15T04:54:02.957 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:54:02.957 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 173 k 2023-11-15T04:54:02.957 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 649 k 2023-11-15T04:54:02.958 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-11-15T04:54:03.030 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 2.3 MB/s | 173 kB 00:00 2023-11-15T04:54:03.031 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:54:03.031 INFO:teuthology.orchestra.run.smithi178.stdout:Total 2.2 MB/s | 173 kB 00:00 2023-11-15T04:54:03.031 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-11-15T04:54:03.041 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-11-15T04:54:03.041 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-11-15T04:54:03.091 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-11-15T04:54:03.094 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-11-15T04:54:03.217 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-11-15T04:54:03.441 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:54:03.730 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:54:04.401 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:54:04.401 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:54:04.401 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-11-15T04:54:04.401 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-15T04:54:04.402 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:54:04.402 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-11-15T04:54:04.632 DEBUG:teuthology.parallel:result is None 2023-11-15T04:54:04.633 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:54:04.633 INFO:teuthology.packaging:ref: None 2023-11-15T04:54:04.633 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:54:04.634 INFO:teuthology.packaging:branch: None 2023-11-15T04:54:04.634 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:54:04.837 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:54:04.838 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:54:05.077 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-15T04:54:05.105 INFO:teuthology.orchestra.run.smithi163.stdout:16.2.4-0.el8 2023-11-15T04:54:05.106 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-15T04:54:05.106 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-15T04:54:05.107 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:54:05.108 INFO:teuthology.packaging:ref: None 2023-11-15T04:54:05.108 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:54:05.108 INFO:teuthology.packaging:branch: None 2023-11-15T04:54:05.109 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:54:05.296 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:54:05.297 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:54:05.519 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-15T04:54:05.545 INFO:teuthology.orchestra.run.smithi178.stdout:16.2.4-0.el8 2023-11-15T04:54:05.546 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-15T04:54:05.546 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-15T04:54:05.547 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-15T04:54:05.547 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-11-15T04:54:05.548 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-15T04:54:05.582 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-15T04:54:05.582 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-15T04:54:05.615 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-15T04:54:05.616 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-11-15T04:54:05.616 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/daemon-helper 2023-11-15T04:54:05.652 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-15T04:54:05.722 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-15T04:54:05.723 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/daemon-helper 2023-11-15T04:54:05.752 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-15T04:54:05.822 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-15T04:54:05.823 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-11-15T04:54:05.823 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-15T04:54:05.853 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-15T04:54:05.922 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-15T04:54:05.923 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-15T04:54:05.951 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-15T04:54:06.019 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-15T04:54:06.020 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-11-15T04:54:06.020 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/stdin-killer 2023-11-15T04:54:06.050 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-15T04:54:06.119 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-15T04:54:06.119 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/stdin-killer 2023-11-15T04:54:06.147 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-15T04:54:06.214 INFO:teuthology.run_tasks:Running task print... 2023-11-15T04:54:06.224 INFO:teuthology.task.print:**** done install task... 2023-11-15T04:54:06.224 INFO:teuthology.run_tasks:Running task cephadm... 2023-11-15T04:54:06.315 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'} 2023-11-15T04:54:06.316 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2023-11-15T04:54:06.317 INFO:tasks.cephadm:Cluster fsid is 01689f7e-8373-11ee-8dbe-212e2dc638e7 2023-11-15T04:54:06.318 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-11-15T04:54:06.318 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-11-15T04:54:06.318 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi163': '172.21.15.163', 'mon.smithi178': '172.21.15.178'} 2023-11-15T04:54:06.318 INFO:tasks.cephadm:Normalizing hostnames... 2023-11-15T04:54:06.319 DEBUG:teuthology.orchestra.run.smithi163:> sudo hostname $(hostname -s) 2023-11-15T04:54:06.349 DEBUG:teuthology.orchestra.run.smithi178:> sudo hostname $(hostname -s) 2023-11-15T04:54:06.377 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2023-11-15T04:54:06.378 DEBUG:teuthology.orchestra.run.smithi163:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-15T04:54:06.618 INFO:teuthology.orchestra.run.smithi163.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 15 04:54 /home/ubuntu/cephtest/cephadm 2023-11-15T04:54:06.619 DEBUG:teuthology.orchestra.run.smithi178:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-15T04:54:06.868 INFO:teuthology.orchestra.run.smithi178.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 15 04:54 /home/ubuntu/cephtest/cephadm 2023-11-15T04:54:06.869 DEBUG:teuthology.orchestra.run.smithi163:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-15T04:54:06.891 DEBUG:teuthology.orchestra.run.smithi178:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-15T04:54:06.921 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /etc/ceph 2023-11-15T04:54:06.964 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph 2023-11-15T04:54:06.991 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 777 /etc/ceph 2023-11-15T04:54:07.037 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 777 /etc/ceph 2023-11-15T04:54:07.064 INFO:tasks.cephadm:Writing seed config... 2023-11-15T04:54:07.065 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-11-15T04:54:07.065 INFO:tasks.cephadm: override: [client] debug client = 20 2023-11-15T04:54:07.066 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-11-15T04:54:07.066 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-11-15T04:54:07.066 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-11-15T04:54:07.067 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-11-15T04:54:07.067 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-11-15T04:54:07.067 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-11-15T04:54:07.068 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-11-15T04:54:07.068 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-11-15T04:54:07.068 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-11-15T04:54:07.069 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-11-15T04:54:07.069 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-11-15T04:54:07.069 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-11-15T04:54:07.069 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-11-15T04:54:07.070 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-11-15T04:54:07.070 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-11-15T04:54:07.070 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-11-15T04:54:07.071 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-11-15T04:54:07.071 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-11-15T04:54:07.072 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-11-15T04:54:07.072 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-11-15T04:54:07.072 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-11-15T04:54:07.072 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-11-15T04:54:07.073 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-11-15T04:54:07.073 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-11-15T04:54:07.074 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-11-15T04:54:07.074 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-11-15T04:54:07.074 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-11-15T04:54:07.074 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-11-15T04:54:07.075 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-11-15T04:54:07.075 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-11-15T04:54:07.075 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-11-15T04:54:07.076 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-11-15T04:54:07.076 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-11-15T04:54:07.076 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-11-15T04:54:07.076 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-11-15T04:54:07.077 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-11-15T04:54:07.078 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-11-15T04:54:07.078 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-11-15T04:54:07.099 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 01689f7e-8373-11ee-8dbe-212e2dc638e7 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-11-15T04:54:07.099 DEBUG:teuthology.orchestra.run.smithi163:mon.smithi163> sudo journalctl -f -n 0 -u ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mon.smithi163.service 2023-11-15T04:54:07.142 INFO:tasks.cephadm:Bootstrapping... 2023-11-15T04:54:07.142 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.163 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-11-15T04:54:07.171 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:-- Logs begin at Wed 2023-11-15 04:41:52 UTC. -- 2023-11-15T04:54:07.404 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T04:54:08.399 INFO:teuthology.orchestra.run.smithi163.stderr:/bin/podman: 4.6.1 2023-11-15T04:54:08.411 INFO:teuthology.orchestra.run.smithi163.stderr:Verifying podman|docker is present... 2023-11-15T04:54:08.412 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T04:54:08.431 INFO:teuthology.orchestra.run.smithi163.stderr:/bin/podman: 4.6.1 2023-11-15T04:54:08.440 INFO:teuthology.orchestra.run.smithi163.stderr:Verifying lvm2 is present... 2023-11-15T04:54:08.441 INFO:teuthology.orchestra.run.smithi163.stderr:Verifying time synchronization is in place... 2023-11-15T04:54:08.441 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl is-enabled chrony.service 2023-11-15T04:54:08.447 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-15T04:54:08.448 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl is-active chrony.service 2023-11-15T04:54:08.454 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: inactive 2023-11-15T04:54:08.454 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-15T04:54:08.460 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: enabled 2023-11-15T04:54:08.461 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl is-active chronyd.service 2023-11-15T04:54:08.466 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: active 2023-11-15T04:54:08.468 INFO:teuthology.orchestra.run.smithi163.stderr:Unit chronyd.service is enabled and running 2023-11-15T04:54:08.468 INFO:teuthology.orchestra.run.smithi163.stderr:Repeating the final host check... 2023-11-15T04:54:08.468 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T04:54:08.487 INFO:teuthology.orchestra.run.smithi163.stderr:/bin/podman: 4.6.1 2023-11-15T04:54:08.497 INFO:teuthology.orchestra.run.smithi163.stderr:podman|docker (/bin/podman) is present 2023-11-15T04:54:08.497 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl is present 2023-11-15T04:54:08.497 INFO:teuthology.orchestra.run.smithi163.stderr:lvcreate is present 2023-11-15T04:54:08.497 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl is-enabled chrony.service 2023-11-15T04:54:08.502 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-15T04:54:08.503 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl is-active chrony.service 2023-11-15T04:54:08.508 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: inactive 2023-11-15T04:54:08.509 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-15T04:54:08.514 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: enabled 2023-11-15T04:54:08.515 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl is-active chronyd.service 2023-11-15T04:54:08.520 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: active 2023-11-15T04:54:08.521 INFO:teuthology.orchestra.run.smithi163.stderr:Unit chronyd.service is enabled and running 2023-11-15T04:54:08.521 INFO:teuthology.orchestra.run.smithi163.stderr:Host looks OK 2023-11-15T04:54:08.522 INFO:teuthology.orchestra.run.smithi163.stderr:Cluster fsid: 01689f7e-8373-11ee-8dbe-212e2dc638e7 2023-11-15T04:54:08.522 INFO:teuthology.orchestra.run.smithi163.stderr:Acquiring lock 140099707816424 on /run/cephadm/01689f7e-8373-11ee-8dbe-212e2dc638e7.lock 2023-11-15T04:54:08.522 INFO:teuthology.orchestra.run.smithi163.stderr:Lock 140099707816424 acquired on /run/cephadm/01689f7e-8373-11ee-8dbe-212e2dc638e7.lock 2023-11-15T04:54:08.523 INFO:teuthology.orchestra.run.smithi163.stderr:Verifying IP 172.21.15.163 port 3300 ... 2023-11-15T04:54:08.523 INFO:teuthology.orchestra.run.smithi163.stderr:Verifying IP 172.21.15.163 port 6789 ... 2023-11-15T04:54:08.523 INFO:teuthology.orchestra.run.smithi163.stderr:Base mon IP is 172.21.15.163, final addrv is [v2:172.21.15.163:3300,v1:172.21.15.163:6789] 2023-11-15T04:54:08.523 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /sbin/ip route ls 2023-11-15T04:54:08.524 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.163 metric 100 2023-11-15T04:54:08.525 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.163 metric 100 2023-11-15T04:54:08.525 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /sbin/ip -6 route ls 2023-11-15T04:54:08.527 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-11-15T04:54:08.528 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2023-11-15T04:54:08.528 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1799sec hoplimit 64 pref medium 2023-11-15T04:54:08.528 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /sbin/ip -6 addr ls 2023-11-15T04:54:08.531 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-11-15T04:54:08.531 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: inet6 ::1/128 scope host 2023-11-15T04:54:08.531 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-15T04:54:08.532 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-11-15T04:54:08.532 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe88:7d5b/64 scope link 2023-11-15T04:54:08.532 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-15T04:54:08.533 INFO:teuthology.orchestra.run.smithi163.stderr:Mon IP 172.21.15.163 is in CIDR network 172.21.0.0/20 2023-11-15T04:54:08.533 INFO:teuthology.orchestra.run.smithi163.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-11-15T04:54:08.533 INFO:teuthology.orchestra.run.smithi163.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2023-11-15T04:54:08.533 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2023-11-15T04:54:08.625 INFO:teuthology.orchestra.run.smithi163.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2023-11-15T04:54:15.154 INFO:teuthology.orchestra.run.smithi163.stderr:/bin/podman: Getting image source signatures 2023-11-15T04:54:15.168 INFO:teuthology.orchestra.run.smithi163.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2023-11-15T04:54:15.169 INFO:teuthology.orchestra.run.smithi163.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2023-11-15T04:54:27.404 INFO:teuthology.orchestra.run.smithi163.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-15T04:54:27.454 INFO:teuthology.orchestra.run.smithi163.stderr:/bin/podman: Writing manifest to image destination 2023-11-15T04:54:27.792 INFO:teuthology.orchestra.run.smithi163.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-15T04:54:27.804 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2023-11-15T04:54:28.396 INFO:teuthology.orchestra.run.smithi163.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-15T04:54:28.747 INFO:teuthology.orchestra.run.smithi163.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-15T04:54:28.747 INFO:teuthology.orchestra.run.smithi163.stderr:Extracting ceph user uid/gid from container image... 2023-11-15T04:54:28.747 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2023-11-15T04:54:29.055 INFO:teuthology.orchestra.run.smithi163.stderr:stat: 167 167 2023-11-15T04:54:29.368 INFO:teuthology.orchestra.run.smithi163.stderr:Creating initial keys... 2023-11-15T04:54:29.368 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T04:54:29.696 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-authtool: AQAFT1RlqAIZJxAAChBV0SZiJd+HugCbOS4q2A== 2023-11-15T04:54:30.030 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T04:54:30.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-authtool: AQAGT1RlORTCEBAAlKKQIIR/KUuYg+RHfgIoZA== 2023-11-15T04:54:30.633 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T04:54:30.920 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-authtool: AQAGT1RlMlIHNhAAIpvUqj7vsQ80ieCC3xwBsQ== 2023-11-15T04:54:31.259 INFO:teuthology.orchestra.run.smithi163.stderr:Creating initial monmap... 2023-11-15T04:54:31.259 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmp6m30lshb:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 --addv smithi163 [v2:172.21.15.163:3300,v1:172.21.15.163:6789] /tmp/monmap 2023-11-15T04:54:31.621 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-15T04:54:31.621 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 01689f7e-8373-11ee-8dbe-212e2dc638e7 2023-11-15T04:54:31.621 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-15T04:54:31.941 INFO:teuthology.orchestra.run.smithi163.stderr:monmaptool for smithi163 [v2:172.21.15.163:3300,v1:172.21.15.163:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-15T04:54:31.941 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/monmaptool: set fsid to 01689f7e-8373-11ee-8dbe-212e2dc638e7 2023-11-15T04:54:31.941 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-15T04:54:31.941 INFO:teuthology.orchestra.run.smithi163.stderr: 2023-11-15T04:54:31.942 INFO:teuthology.orchestra.run.smithi163.stderr:Creating mon... 2023-11-15T04:54:31.942 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/mon.smithi163:/var/lib/ceph/mon/ceph-smithi163:z -v /tmp/ceph-tmp_g5e64ya:/tmp/keyring:z -v /tmp/ceph-tmp6m30lshb:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi163 --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2023-11-15T04:54:32.244 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.210+0000 7f76875b7700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-15T04:54:32.245 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.210+0000 7f76875b7700 1 imported monmap: 2023-11-15T04:54:32.245 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: epoch 0 2023-11-15T04:54:32.245 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 2023-11-15T04:54:32.245 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: last_changed 2023-11-15T04:54:31.569260+0000 2023-11-15T04:54:32.246 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: created 2023-11-15T04:54:31.569260+0000 2023-11-15T04:54:32.246 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-11-15T04:54:32.246 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-11-15T04:54:32.246 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.smithi163 2023-11-15T04:54:32.246 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2023-11-15T04:54:32.247 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.210+0000 7f76875b7700 0 /usr/bin/ceph-mon: set fsid to 01689f7e-8373-11ee-8dbe-212e2dc638e7 2023-11-15T04:54:32.247 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: RocksDB version: 6.8.1 2023-11-15T04:54:32.247 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2023-11-15T04:54:32.247 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-15T04:54:32.247 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Compile date May 13 2021 2023-11-15T04:54:32.247 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: DB SUMMARY 2023-11-15T04:54:32.248 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2023-11-15T04:54:32.248 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi163/store.db dir, Total Num: 0, files: 2023-11-15T04:54:32.248 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2023-11-15T04:54:32.248 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi163/store.db: 2023-11-15T04:54:32.248 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2023-11-15T04:54:32.249 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.error_if_exists: 0 2023-11-15T04:54:32.249 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.create_if_missing: 1 2023-11-15T04:54:32.249 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.paranoid_checks: 1 2023-11-15T04:54:32.249 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.env: 0x5642444851c0 2023-11-15T04:54:32.249 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.fs: Posix File System 2023-11-15T04:54:32.250 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.info_log: 0x5642457b6ec0 2023-11-15T04:54:32.250 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-15T04:54:32.250 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.statistics: (nil) 2023-11-15T04:54:32.250 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.use_fsync: 0 2023-11-15T04:54:32.250 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.max_log_file_size: 0 2023-11-15T04:54:32.251 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-15T04:54:32.251 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-15T04:54:32.251 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-15T04:54:32.251 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-15T04:54:32.251 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.allow_fallocate: 1 2023-11-15T04:54:32.252 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-15T04:54:32.252 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-15T04:54:32.252 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.use_direct_reads: 0 2023-11-15T04:54:32.252 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-15T04:54:32.252 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-15T04:54:32.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.db_log_dir: 2023-11-15T04:54:32.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi163/store.db 2023-11-15T04:54:32.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-15T04:54:32.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.max_subcompactions: 1 2023-11-15T04:54:32.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.max_background_flushes: -1 2023-11-15T04:54:32.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-15T04:54:32.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-15T04:54:32.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-15T04:54:32.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-15T04:54:32.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-15T04:54:32.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-15T04:54:32.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-15T04:54:32.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.write_buffer_manager: 0x5642457bd530 2023-11-15T04:54:32.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-15T04:54:32.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-15T04:54:32.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-15T04:54:32.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-15T04:54:32.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-15T04:54:32.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-15T04:54:32.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-15T04:54:32.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-15T04:54:32.257 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-15T04:54:32.257 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.unordered_write: 0 2023-11-15T04:54:32.257 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-15T04:54:32.257 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-15T04:54:32.257 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-15T04:54:32.258 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-15T04:54:32.258 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.row_cache: None 2023-11-15T04:54:32.258 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.wal_filter: None 2023-11-15T04:54:32.258 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-15T04:54:32.258 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-15T04:54:32.259 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.preserve_deletes: 0 2023-11-15T04:54:32.259 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.two_write_queues: 0 2023-11-15T04:54:32.259 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-15T04:54:32.259 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.atomic_flush: 0 2023-11-15T04:54:32.259 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-15T04:54:32.259 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-15T04:54:32.260 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-15T04:54:32.260 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.log_readahead_size: 0 2023-11-15T04:54:32.260 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-15T04:54:32.260 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.max_background_jobs: 2 2023-11-15T04:54:32.260 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.max_background_compactions: -1 2023-11-15T04:54:32.261 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-15T04:54:32.261 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-15T04:54:32.261 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-15T04:54:32.261 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-15T04:54:32.261 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-15T04:54:32.262 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-15T04:54:32.262 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-15T04:54:32.262 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-15T04:54:32.262 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.max_open_files: -1 2023-11-15T04:54:32.262 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-15T04:54:32.263 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-15T04:54:32.263 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-15T04:54:32.263 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-15T04:54:32.263 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Compression algorithms supported: 2023-11-15T04:54:32.263 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-15T04:54:32.264 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: kZSTD supported: 0 2023-11-15T04:54:32.264 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: kXpressCompression supported: 0 2023-11-15T04:54:32.264 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-15T04:54:32.264 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: kLZ4Compression supported: 1 2023-11-15T04:54:32.264 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: kBZip2Compression supported: 0 2023-11-15T04:54:32.264 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: kZlibCompression supported: 1 2023-11-15T04:54:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: kSnappyCompression supported: 1 2023-11-15T04:54:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-15T04:54:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.211+0000 7f76875b7700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-11-15T04:54:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2023-11-15T04:54:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi163/store.db/MANIFEST-000001 2023-11-15T04:54:32.266 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2023-11-15T04:54:32.266 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-15T04:54:32.266 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2023-11-15T04:54:32.266 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-15T04:54:32.266 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.merge_operator: 2023-11-15T04:54:32.267 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.compaction_filter: None 2023-11-15T04:54:32.267 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-15T04:54:32.267 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-15T04:54:32.267 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-15T04:54:32.267 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5642456ce120) 2023-11-15T04:54:32.268 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-11-15T04:54:32.268 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-15T04:54:32.268 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-15T04:54:32.268 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-11-15T04:54:32.268 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: index_type: 0 2023-11-15T04:54:32.268 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-11-15T04:54:32.269 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-11-15T04:54:32.269 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-11-15T04:54:32.269 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-11-15T04:54:32.269 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: checksum: 1 2023-11-15T04:54:32.269 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-11-15T04:54:32.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: block_cache: 0x564245704d10 2023-11-15T04:54:32.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-11-15T04:54:32.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-11-15T04:54:32.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-11-15T04:54:32.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-11-15T04:54:32.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-11-15T04:54:32.271 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-11-15T04:54:32.271 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-11-15T04:54:32.271 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-11-15T04:54:32.271 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-11-15T04:54:32.271 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-11-15T04:54:32.272 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-11-15T04:54:32.272 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-11-15T04:54:32.272 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-11-15T04:54:32.272 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-11-15T04:54:32.272 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-11-15T04:54:32.273 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-15T04:54:32.273 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-11-15T04:54:32.273 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-11-15T04:54:32.273 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-11-15T04:54:32.273 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: format_version: 2 2023-11-15T04:54:32.274 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-11-15T04:54:32.274 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: block_align: 0 2023-11-15T04:54:32.274 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2023-11-15T04:54:32.274 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-15T04:54:32.274 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-15T04:54:32.275 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.compression: NoCompression 2023-11-15T04:54:32.275 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-15T04:54:32.275 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-15T04:54:32.275 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-15T04:54:32.275 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.num_levels: 7 2023-11-15T04:54:32.276 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-15T04:54:32.276 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-15T04:54:32.276 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-15T04:54:32.276 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-15T04:54:32.276 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-15T04:54:32.276 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-15T04:54:32.277 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-15T04:54:32.277 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-15T04:54:32.277 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-15T04:54:32.277 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-15T04:54:32.277 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-15T04:54:32.278 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-15T04:54:32.278 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-15T04:54:32.278 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-15T04:54:32.278 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-15T04:54:32.278 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-15T04:54:32.279 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-15T04:54:32.279 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-15T04:54:32.279 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-15T04:54:32.279 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-15T04:54:32.279 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-15T04:54:32.280 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-15T04:54:32.280 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-15T04:54:32.280 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-15T04:54:32.280 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-15T04:54:32.280 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-15T04:54:32.280 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-15T04:54:32.281 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-15T04:54:32.281 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-15T04:54:32.281 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-15T04:54:32.281 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-15T04:54:32.281 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-15T04:54:32.281 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-15T04:54:32.282 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-15T04:54:32.282 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-15T04:54:32.282 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-15T04:54:32.282 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-15T04:54:32.282 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-15T04:54:32.283 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-15T04:54:32.283 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-15T04:54:32.283 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-15T04:54:32.283 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-15T04:54:32.283 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-15T04:54:32.284 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-15T04:54:32.284 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-15T04:54:32.284 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-15T04:54:32.284 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-15T04:54:32.284 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.table_properties_collectors: 2023-11-15T04:54:32.284 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.inplace_update_support: 0 2023-11-15T04:54:32.285 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-15T04:54:32.285 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-15T04:54:32.285 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-15T04:54:32.285 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-15T04:54:32.286 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.bloom_locality: 0 2023-11-15T04:54:32.286 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.max_successive_merges: 0 2023-11-15T04:54:32.286 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-15T04:54:32.286 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-15T04:54:32.286 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-15T04:54:32.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-15T04:54:32.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.ttl: 2592000 2023-11-15T04:54:32.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-15T04:54:32.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi163/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-15T04:54:32.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2023-11-15T04:54:32.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.212+0000 7f76875b7700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-11-15T04:54:32.288 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2023-11-15T04:54:32.288 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.213+0000 7f76875b7700 4 rocksdb: DB pointer 0x5642457c9800 2023-11-15T04:54:32.288 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.213+0000 7f7670450700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-15T04:54:32.288 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.213+0000 7f7670450700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-15T04:54:32.288 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-11-15T04:54:32.289 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T04:54:32.289 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-15T04:54:32.289 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-15T04:54:32.289 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T04:54:32.289 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-15T04:54:32.290 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-15T04:54:32.290 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T04:54:32.290 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2023-11-15T04:54:32.290 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T04:54:32.290 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:54:32.291 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:54:32.291 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T04:54:32.291 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T04:54:32.291 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2023-11-15T04:54:32.291 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T04:54:32.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:54:32.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:54:32.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T04:54:32.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T04:54:32.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T04:54:32.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T04:54:32.293 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T04:54:32.293 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-15T04:54:32.293 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:54:32.293 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:54:32.293 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T04:54:32.294 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2023-11-15T04:54:32.294 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-15T04:54:32.294 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2023-11-15T04:54:32.294 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T04:54:32.294 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:54:32.294 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:54:32.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T04:54:32.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T04:54:32.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2023-11-15T04:54:32.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T04:54:32.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:54:32.296 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:54:32.296 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T04:54:32.296 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T04:54:32.296 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T04:54:32.296 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T04:54:32.297 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T04:54:32.297 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-15T04:54:32.297 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:54:32.297 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:54:32.297 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T04:54:32.297 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2023-11-15T04:54:32.298 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-15T04:54:32.298 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2023-11-15T04:54:32.298 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.214+0000 7f76875b7700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-11-15T04:54:32.298 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.214+0000 7f76875b7700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-11-15T04:54:32.299 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2023-11-15T04:54:32.214+0000 7f76875b7700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi163 for mon.smithi163 2023-11-15T04:54:32.589 INFO:teuthology.orchestra.run.smithi163.stderr:create mon.smithi163 on 2023-11-15T04:54:32.589 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7 2023-11-15T04:54:32.593 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl disable ceph.target 2023-11-15T04:54:32.600 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-11-15T04:54:32.683 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl enable ceph.target 2023-11-15T04:54:32.691 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-11-15T04:54:32.771 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl start ceph.target 2023-11-15T04:54:32.778 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl enable ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7.target 2023-11-15T04:54:32.784 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7.target. 2023-11-15T04:54:32.784 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7.target. 2023-11-15T04:54:32.862 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl start ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7.target 2023-11-15T04:54:32.869 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl daemon-reload 2023-11-15T04:54:32.951 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl stop ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mon.smithi163 2023-11-15T04:54:32.981 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl reset-failed ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mon.smithi163 2023-11-15T04:54:32.987 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: Failed to reset failed state of unit ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mon.smithi163.service: Unit ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mon.smithi163.service not loaded. 2023-11-15T04:54:32.988 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl enable ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mon.smithi163 2023-11-15T04:54:32.994 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: Created symlink /etc/systemd/system/ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7.target.wants/ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mon.smithi163.service → /etc/systemd/system/ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@.service. 2023-11-15T04:54:33.079 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl start ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mon.smithi163 2023-11-15T04:54:33.119 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:33 smithi163 systemd[1]: Starting Ceph mon.smithi163 for 01689f7e-8373-11ee-8dbe-212e2dc638e7... 2023-11-15T04:54:33.432 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T04:54:33.441 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: disabled 2023-11-15T04:54:33.442 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T04:54:33.448 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: inactive 2023-11-15T04:54:33.449 INFO:teuthology.orchestra.run.smithi163.stderr:firewalld.service is not enabled 2023-11-15T04:54:33.450 INFO:teuthology.orchestra.run.smithi163.stderr:Not possible to enable service . firewalld.service is not available 2023-11-15T04:54:33.450 INFO:teuthology.orchestra.run.smithi163.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-15T04:54:33.450 INFO:teuthology.orchestra.run.smithi163.stderr:Waiting for mon to start... 2023-11-15T04:54:33.451 INFO:teuthology.orchestra.run.smithi163.stderr:Waiting for mon... 2023-11-15T04:54:33.451 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/mon.smithi163:/var/lib/ceph/mon/ceph-smithi163:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2023-11-15T04:54:33.703 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:33 smithi163 bash[111922]: 2af02a466534d5a42fd8b0a5118a2a7d786c97b00dc40b46bb073511d19708ba 2023-11-15T04:54:33.703 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:33 smithi163 systemd[1]: Started Ceph mon.smithi163 for 01689f7e-8373-11ee-8dbe-212e2dc638e7. 2023-11-15T04:54:33.703 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:33 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[111936]: cluster 2023-11-15T04:54:33.468087+0000 mon.smithi163 (mon.0) 0 : [INF] mkfs 01689f7e-8373-11ee-8dbe-212e2dc638e7 2023-11-15T04:54:33.703 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:33 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[111936]: cluster 2023-11-15T04:54:33.449066+0000 mon.smithi163 (mon.0) 1 : cluster [INF] mon.smithi163 is new leader, mons smithi163 in quorum (ranks 0) 2023-11-15T04:54:33.851 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.848+0000 7f1555747700 1 Processor -- start 2023-11-15T04:54:33.851 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.848+0000 7f1555747700 1 -- start start 2023-11-15T04:54:33.852 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.848+0000 7f1555747700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15500fc2c0 0x7f15500fc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:33.852 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.848+0000 7f1555747700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15500fcbd0 con 0x7f15500fc2c0 2023-11-15T04:54:33.852 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.849+0000 7f154effd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15500fc2c0 0x7f15500fc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:33.852 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.849+0000 7f154effd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15500fc2c0 0x7f15500fc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40122/0 (socket says 172.21.15.163:40122) 2023-11-15T04:54:33.852 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.849+0000 7f154effd700 1 -- 172.21.15.163:0/4222897453 learned_addr learned my addr 172.21.15.163:0/4222897453 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:33.853 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.849+0000 7f154effd700 1 -- 172.21.15.163:0/4222897453 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15500fcd10 con 0x7f15500fc2c0 2023-11-15T04:54:33.853 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.849+0000 7f154effd700 1 --2- 172.21.15.163:0/4222897453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15500fc2c0 0x7f15500fc6e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f154400b5f0 tx=0x7f1544006ee0).ready entity=mon.0 client_cookie=41ca316b7bbd95a8 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:33.853 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.849+0000 7f154dffb700 1 -- 172.21.15.163:0/4222897453 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f154400d600 con 0x7f15500fc2c0 2023-11-15T04:54:33.854 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.849+0000 7f154dffb700 1 -- 172.21.15.163:0/4222897453 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154400d760 con 0x7f15500fc2c0 2023-11-15T04:54:33.854 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.849+0000 7f154dffb700 1 -- 172.21.15.163:0/4222897453 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f154400da30 con 0x7f15500fc2c0 2023-11-15T04:54:33.857 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.850+0000 7f1555747700 1 -- 172.21.15.163:0/4222897453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15500fc2c0 msgr2=0x7f15500fc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:33.858 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.850+0000 7f1555747700 1 --2- 172.21.15.163:0/4222897453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15500fc2c0 0x7f15500fc6e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f154400b5f0 tx=0x7f1544006ee0).stop 2023-11-15T04:54:33.858 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.850+0000 7f1555747700 1 -- 172.21.15.163:0/4222897453 shutdown_connections 2023-11-15T04:54:33.858 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.850+0000 7f1555747700 1 --2- 172.21.15.163:0/4222897453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15500fc2c0 0x7f15500fc6e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:33.858 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.850+0000 7f1555747700 1 -- 172.21.15.163:0/4222897453 >> 172.21.15.163:0/4222897453 conn(0x7f15500f5b10 msgr2=0x7f15500f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:33.859 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.850+0000 7f1555747700 1 -- 172.21.15.163:0/4222897453 shutdown_connections 2023-11-15T04:54:33.859 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.850+0000 7f1555747700 1 -- 172.21.15.163:0/4222897453 wait complete. 2023-11-15T04:54:33.859 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.851+0000 7f1555747700 1 Processor -- start 2023-11-15T04:54:33.860 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.851+0000 7f1555747700 1 -- start start 2023-11-15T04:54:33.860 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.851+0000 7f1555747700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15500fc2c0 0x7f155010c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:33.860 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.851+0000 7f1555747700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155010c970 con 0x7f15500fc2c0 2023-11-15T04:54:33.861 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.851+0000 7f154effd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15500fc2c0 0x7f155010c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:33.861 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.851+0000 7f154effd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15500fc2c0 0x7f155010c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40124/0 (socket says 172.21.15.163:40124) 2023-11-15T04:54:33.861 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.851+0000 7f154effd700 1 -- 172.21.15.163:0/1032864538 learned_addr learned my addr 172.21.15.163:0/1032864538 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:33.861 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.851+0000 7f154effd700 1 -- 172.21.15.163:0/1032864538 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154400b200 con 0x7f15500fc2c0 2023-11-15T04:54:33.861 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.851+0000 7f154effd700 1 --2- 172.21.15.163:0/1032864538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15500fc2c0 0x7f155010c480 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f1544006c70 tx=0x7f1544016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:33.862 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.852+0000 7f15377fe700 1 -- 172.21.15.163:0/1032864538 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1544011b40 con 0x7f15500fc2c0 2023-11-15T04:54:33.862 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.852+0000 7f15377fe700 1 -- 172.21.15.163:0/1032864538 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1544011ca0 con 0x7f15500fc2c0 2023-11-15T04:54:33.862 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.852+0000 7f1555747700 1 -- 172.21.15.163:0/1032864538 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f155010cb70 con 0x7f15500fc2c0 2023-11-15T04:54:33.862 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.852+0000 7f15377fe700 1 -- 172.21.15.163:0/1032864538 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f154401d430 con 0x7f15500fc2c0 2023-11-15T04:54:33.863 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.852+0000 7f1555747700 1 -- 172.21.15.163:0/1032864538 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f155010cf80 con 0x7f15500fc2c0 2023-11-15T04:54:33.863 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.852+0000 7f15377fe700 1 -- 172.21.15.163:0/1032864538 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f154400f070 con 0x7f15500fc2c0 2023-11-15T04:54:33.863 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.852+0000 7f15377fe700 1 -- 172.21.15.163:0/1032864538 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1544027460 con 0x7f15500fc2c0 2023-11-15T04:54:33.863 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.853+0000 7f1555747700 1 -- 172.21.15.163:0/1032864538 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1550045ad0 con 0x7f15500fc2c0 2023-11-15T04:54:33.863 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.855+0000 7f15377fe700 1 -- 172.21.15.163:0/1032864538 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f1544016070 con 0x7f15500fc2c0 2023-11-15T04:54:33.927 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.921+0000 7f1555747700 1 -- 172.21.15.163:0/1032864538 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f1550106010 con 0x7f15500fc2c0 2023-11-15T04:54:33.927 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.921+0000 7f15377fe700 1 -- 172.21.15.163:0/1032864538 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f154401d640 con 0x7f15500fc2c0 2023-11-15T04:54:33.928 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: cluster: 2023-11-15T04:54:33.928 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: id: 01689f7e-8373-11ee-8dbe-212e2dc638e7 2023-11-15T04:54:33.929 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-11-15T04:54:33.930 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.930 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: services: 2023-11-15T04:54:33.930 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi163 (age 0.456224s) 2023-11-15T04:54:33.930 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: mgr: no daemons active 2023-11-15T04:54:33.931 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-11-15T04:54:33.931 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.931 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: data: 2023-11-15T04:54:33.931 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-11-15T04:54:33.931 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-11-15T04:54:33.932 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-11-15T04:54:33.932 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: pgs: 2023-11-15T04:54:33.932 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.932 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.922+0000 7f1555747700 1 -- 172.21.15.163:0/1032864538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15500fc2c0 msgr2=0x7f155010c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:33.932 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.922+0000 7f1555747700 1 --2- 172.21.15.163:0/1032864538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15500fc2c0 0x7f155010c480 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f1544006c70 tx=0x7f1544016000).stop 2023-11-15T04:54:33.932 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.922+0000 7f1555747700 1 -- 172.21.15.163:0/1032864538 shutdown_connections 2023-11-15T04:54:33.933 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.922+0000 7f1555747700 1 --2- 172.21.15.163:0/1032864538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15500fc2c0 0x7f155010c480 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:33.933 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.922+0000 7f1555747700 1 -- 172.21.15.163:0/1032864538 >> 172.21.15.163:0/1032864538 conn(0x7f15500f5b10 msgr2=0x7f15500f7800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:33.933 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.922+0000 7f1555747700 1 -- 172.21.15.163:0/1032864538 shutdown_connections 2023-11-15T04:54:33.933 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:33.922+0000 7f1555747700 1 -- 172.21.15.163:0/1032864538 wait complete. 2023-11-15T04:54:34.264 INFO:teuthology.orchestra.run.smithi163.stderr:mon is available 2023-11-15T04:54:34.264 INFO:teuthology.orchestra.run.smithi163.stderr:Assimilating anything we can from ceph.conf... 2023-11-15T04:54:34.264 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z -v /var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/mon.smithi163:/var/lib/ceph/mon/ceph-smithi163:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi163/config 2023-11-15T04:54:34.642 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.639+0000 7f81b7169700 1 Processor -- start 2023-11-15T04:54:34.643 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.639+0000 7f81b7169700 1 -- start start 2023-11-15T04:54:34.643 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.639+0000 7f81b7169700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81b00fbca0 0x7f81b00fc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:34.643 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.639+0000 7f81b7169700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b00fc5b0 con 0x7f81b00fbca0 2023-11-15T04:54:34.643 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.639+0000 7f81b4f05700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81b00fbca0 0x7f81b00fc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:34.643 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.639+0000 7f81b4f05700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81b00fbca0 0x7f81b00fc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40132/0 (socket says 172.21.15.163:40132) 2023-11-15T04:54:34.644 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.639+0000 7f81b4f05700 1 -- 172.21.15.163:0/2027137752 learned_addr learned my addr 172.21.15.163:0/2027137752 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:34.644 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.640+0000 7f81b4f05700 1 -- 172.21.15.163:0/2027137752 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b00fc6f0 con 0x7f81b00fbca0 2023-11-15T04:54:34.644 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.640+0000 7f81b4f05700 1 --2- 172.21.15.163:0/2027137752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81b00fbca0 0x7f81b00fc0c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f81a400b5f0 tx=0x7f81a4006ee0).ready entity=mon.0 client_cookie=450c48223e778d40 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:34.644 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.640+0000 7f81af7fe700 1 -- 172.21.15.163:0/2027137752 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81a400d600 con 0x7f81b00fbca0 2023-11-15T04:54:34.644 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.640+0000 7f81af7fe700 1 -- 172.21.15.163:0/2027137752 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a400d760 con 0x7f81b00fbca0 2023-11-15T04:54:34.645 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.640+0000 7f81af7fe700 1 -- 172.21.15.163:0/2027137752 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81a400da30 con 0x7f81b00fbca0 2023-11-15T04:54:34.645 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.640+0000 7f81b7169700 1 -- 172.21.15.163:0/2027137752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81b00fbca0 msgr2=0x7f81b00fc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:34.645 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.640+0000 7f81b7169700 1 --2- 172.21.15.163:0/2027137752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81b00fbca0 0x7f81b00fc0c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f81a400b5f0 tx=0x7f81a4006ee0).stop 2023-11-15T04:54:34.645 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.640+0000 7f81b7169700 1 -- 172.21.15.163:0/2027137752 shutdown_connections 2023-11-15T04:54:34.645 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.640+0000 7f81b7169700 1 --2- 172.21.15.163:0/2027137752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81b00fbca0 0x7f81b00fc0c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:34.646 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.640+0000 7f81b7169700 1 -- 172.21.15.163:0/2027137752 >> 172.21.15.163:0/2027137752 conn(0x7f81b00f74c0 msgr2=0x7f81b00f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:34.646 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.640+0000 7f81b7169700 1 -- 172.21.15.163:0/2027137752 shutdown_connections 2023-11-15T04:54:34.646 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.640+0000 7f81b7169700 1 -- 172.21.15.163:0/2027137752 wait complete. 2023-11-15T04:54:34.646 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.641+0000 7f81b7169700 1 Processor -- start 2023-11-15T04:54:34.646 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.641+0000 7f81b7169700 1 -- start start 2023-11-15T04:54:34.648 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.642+0000 7f81b7169700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81b00fbca0 0x7f81b010a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:34.648 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.642+0000 7f81b7169700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b010a760 con 0x7f81b00fbca0 2023-11-15T04:54:34.648 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.642+0000 7f81b4f05700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81b00fbca0 0x7f81b010a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:34.648 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.642+0000 7f81b4f05700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81b00fbca0 0x7f81b010a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40144/0 (socket says 172.21.15.163:40144) 2023-11-15T04:54:34.650 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.642+0000 7f81b4f05700 1 -- 172.21.15.163:0/1503331825 learned_addr learned my addr 172.21.15.163:0/1503331825 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:34.650 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.642+0000 7f81b4f05700 1 -- 172.21.15.163:0/1503331825 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a400b200 con 0x7f81b00fbca0 2023-11-15T04:54:34.650 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.642+0000 7f81b4f05700 1 --2- 172.21.15.163:0/1503331825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81b00fbca0 0x7f81b010a270 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f81a4006c70 tx=0x7f81a4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:34.650 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.642+0000 7f81ad7fa700 1 -- 172.21.15.163:0/1503331825 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81a4011b40 con 0x7f81b00fbca0 2023-11-15T04:54:34.651 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.642+0000 7f81ad7fa700 1 -- 172.21.15.163:0/1503331825 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a4011ca0 con 0x7f81b00fbca0 2023-11-15T04:54:34.651 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.642+0000 7f81ad7fa700 1 -- 172.21.15.163:0/1503331825 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81a401d430 con 0x7f81b00fbca0 2023-11-15T04:54:34.651 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.642+0000 7f81b7169700 1 -- 172.21.15.163:0/1503331825 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b010a960 con 0x7f81b00fbca0 2023-11-15T04:54:34.651 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.642+0000 7f81b7169700 1 -- 172.21.15.163:0/1503331825 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b010ad70 con 0x7f81b00fbca0 2023-11-15T04:54:34.651 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.642+0000 7f81ad7fa700 1 -- 172.21.15.163:0/1503331825 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f81a400f070 con 0x7f81b00fbca0 2023-11-15T04:54:34.652 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.642+0000 7f81ad7fa700 1 -- 172.21.15.163:0/1503331825 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f81a4027460 con 0x7f81b00fbca0 2023-11-15T04:54:34.652 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.643+0000 7f81b7169700 1 -- 172.21.15.163:0/1503331825 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b0045ad0 con 0x7f81b00fbca0 2023-11-15T04:54:34.652 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.645+0000 7f81ad7fa700 1 -- 172.21.15.163:0/1503331825 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f81a4016070 con 0x7f81b00fbca0 2023-11-15T04:54:34.703 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:34 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[111936]: cluster 2023-11-15T04:54:33.466844+0000 mon.smithi163 (mon.0) 2 : cluster [INF] mon.smithi163 is new leader, mons smithi163 in quorum (ranks 0) 2023-11-15T04:54:34.703 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:34 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[111936]: cluster 2023-11-15T04:54:33.467081+0000 mon.smithi163 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi163=[v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0]} 2023-11-15T04:54:34.704 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:34 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[111936]: cluster 2023-11-15T04:54:33.470254+0000 mon.smithi163 (mon.0) 4 : cluster [DBG] fsmap 2023-11-15T04:54:34.704 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:34 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[111936]: cluster 2023-11-15T04:54:33.473801+0000 mon.smithi163 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-15T04:54:34.704 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:34 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[111936]: cluster 2023-11-15T04:54:33.474313+0000 mon.smithi163 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-11-15T04:54:34.704 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:34 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[111936]: audit 2023-11-15T04:54:33.923035+0000 mon.smithi163 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.163:0/1032864538' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-11-15T04:54:34.711 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.710+0000 7f81b7169700 1 -- 172.21.15.163:0/1503331825 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f81b00fcfe0 con 0x7f81b00fbca0 2023-11-15T04:54:34.717 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.714+0000 7f81ad7fa700 1 -- 172.21.15.163:0/1503331825 <== mon.0 v2:172.21.15.163:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f81a401d620 con 0x7f81b00fbca0 2023-11-15T04:54:34.717 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.714+0000 7f81ad7fa700 1 -- 172.21.15.163:0/1503331825 <== mon.0 v2:172.21.15.163:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+466 (secure 0 0 0) 0x7f81a4030830 con 0x7f81b00fbca0 2023-11-15T04:54:34.718 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.718 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: [global] 2023-11-15T04:54:34.718 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: fsid = 01689f7e-8373-11ee-8dbe-212e2dc638e7 2023-11-15T04:54:34.718 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.163:3300,v1:172.21.15.163:6789] 2023-11-15T04:54:34.718 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-11-15T04:54:34.719 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-11-15T04:54:34.719 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-11-15T04:54:34.719 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-11-15T04:54:34.719 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.720 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: [mgr] 2023-11-15T04:54:34.720 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-11-15T04:54:34.720 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.720 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: [osd] 2023-11-15T04:54:34.720 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-11-15T04:54:34.721 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-11-15T04:54:34.721 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-11-15T04:54:34.721 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-11-15T04:54:34.721 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2023-11-15T04:54:34.723 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-11-15T04:54:34.723 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.715+0000 7f81b7169700 1 -- 172.21.15.163:0/1503331825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81b00fbca0 msgr2=0x7f81b010a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:34.723 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.715+0000 7f81b7169700 1 --2- 172.21.15.163:0/1503331825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81b00fbca0 0x7f81b010a270 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f81a4006c70 tx=0x7f81a4016000).stop 2023-11-15T04:54:34.723 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.715+0000 7f81b7169700 1 -- 172.21.15.163:0/1503331825 shutdown_connections 2023-11-15T04:54:34.724 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.715+0000 7f81b7169700 1 --2- 172.21.15.163:0/1503331825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81b00fbca0 0x7f81b010a270 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:34.724 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.715+0000 7f81b7169700 1 -- 172.21.15.163:0/1503331825 >> 172.21.15.163:0/1503331825 conn(0x7f81b00f74c0 msgr2=0x7f81b00fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:34.724 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.715+0000 7f81b7169700 1 -- 172.21.15.163:0/1503331825 shutdown_connections 2023-11-15T04:54:34.724 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:34.715+0000 7f81b7169700 1 -- 172.21.15.163:0/1503331825 wait complete. 2023-11-15T04:54:35.055 INFO:teuthology.orchestra.run.smithi163.stderr:Generating new minimal ceph.conf... 2023-11-15T04:54:35.055 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z -v /var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/mon.smithi163:/var/lib/ceph/mon/ceph-smithi163:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi163/config 2023-11-15T04:54:35.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.439+0000 7f5b60a05700 1 Processor -- start 2023-11-15T04:54:35.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.439+0000 7f5b60a05700 1 -- start start 2023-11-15T04:54:35.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.439+0000 7f5b60a05700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b5c070430 0x7f5b5c0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:35.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.439+0000 7f5b60a05700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c0fedd0 con 0x7f5b5c070430 2023-11-15T04:54:35.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.439+0000 7f5b5a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b5c070430 0x7f5b5c0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:35.448 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.440+0000 7f5b5a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b5c070430 0x7f5b5c0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40152/0 (socket says 172.21.15.163:40152) 2023-11-15T04:54:35.448 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.440+0000 7f5b5a59c700 1 -- 172.21.15.163:0/2573570873 learned_addr learned my addr 172.21.15.163:0/2573570873 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:35.448 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.440+0000 7f5b5a59c700 1 -- 172.21.15.163:0/2573570873 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5c0fef10 con 0x7f5b5c070430 2023-11-15T04:54:35.449 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.440+0000 7f5b5a59c700 1 --2- 172.21.15.163:0/2573570873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b5c070430 0x7f5b5c0fe8e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f5b5000b5f0 tx=0x7f5b50006ee0).ready entity=mon.0 client_cookie=60982fe04c7f2213 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:35.449 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.440+0000 7f5b5959a700 1 -- 172.21.15.163:0/2573570873 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b5000d600 con 0x7f5b5c070430 2023-11-15T04:54:35.449 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.440+0000 7f5b5959a700 1 -- 172.21.15.163:0/2573570873 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5b5000dbd0 con 0x7f5b5c070430 2023-11-15T04:54:35.450 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.440+0000 7f5b5959a700 1 -- 172.21.15.163:0/2573570873 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b50011c00 con 0x7f5b5c070430 2023-11-15T04:54:35.450 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.441+0000 7f5b60a05700 1 -- 172.21.15.163:0/2573570873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b5c070430 msgr2=0x7f5b5c0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:35.450 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.441+0000 7f5b60a05700 1 --2- 172.21.15.163:0/2573570873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b5c070430 0x7f5b5c0fe8e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f5b5000b5f0 tx=0x7f5b50006ee0).stop 2023-11-15T04:54:35.451 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.441+0000 7f5b60a05700 1 -- 172.21.15.163:0/2573570873 shutdown_connections 2023-11-15T04:54:35.451 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.441+0000 7f5b60a05700 1 --2- 172.21.15.163:0/2573570873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b5c070430 0x7f5b5c0fe8e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:35.452 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.441+0000 7f5b60a05700 1 -- 172.21.15.163:0/2573570873 >> 172.21.15.163:0/2573570873 conn(0x7f5b5c0f63c0 msgr2=0x7f5b5c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:35.453 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.441+0000 7f5b60a05700 1 -- 172.21.15.163:0/2573570873 shutdown_connections 2023-11-15T04:54:35.453 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.441+0000 7f5b60a05700 1 -- 172.21.15.163:0/2573570873 wait complete. 2023-11-15T04:54:35.454 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.441+0000 7f5b60a05700 1 Processor -- start 2023-11-15T04:54:35.454 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.441+0000 7f5b60a05700 1 -- start start 2023-11-15T04:54:35.454 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.442+0000 7f5b60a05700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b5c070430 0x7f5b5c0fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.442+0000 7f5b60a05700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c0fa7e0 con 0x7f5b5c070430 2023-11-15T04:54:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.442+0000 7f5b5a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b5c070430 0x7f5b5c0fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:35.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.442+0000 7f5b5a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b5c070430 0x7f5b5c0fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40160/0 (socket says 172.21.15.163:40160) 2023-11-15T04:54:35.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.442+0000 7f5b5a59c700 1 -- 172.21.15.163:0/2900902744 learned_addr learned my addr 172.21.15.163:0/2900902744 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:35.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.442+0000 7f5b5a59c700 1 -- 172.21.15.163:0/2900902744 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5000b200 con 0x7f5b5c070430 2023-11-15T04:54:35.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.442+0000 7f5b5a59c700 1 --2- 172.21.15.163:0/2900902744 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b5c070430 0x7f5b5c0fc360 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5b5000b190 tx=0x7f5b50006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:35.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.442+0000 7f5b4affd700 1 -- 172.21.15.163:0/2900902744 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b5001b410 con 0x7f5b5c070430 2023-11-15T04:54:35.458 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.442+0000 7f5b4affd700 1 -- 172.21.15.163:0/2900902744 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5b5001b9e0 con 0x7f5b5c070430 2023-11-15T04:54:35.458 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.442+0000 7f5b60a05700 1 -- 172.21.15.163:0/2900902744 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b5c0fa980 con 0x7f5b5c070430 2023-11-15T04:54:35.458 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.442+0000 7f5b4affd700 1 -- 172.21.15.163:0/2900902744 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b5001a880 con 0x7f5b5c070430 2023-11-15T04:54:35.459 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.442+0000 7f5b60a05700 1 -- 172.21.15.163:0/2900902744 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b5c0fad90 con 0x7f5b5c070430 2023-11-15T04:54:35.460 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.442+0000 7f5b4affd700 1 -- 172.21.15.163:0/2900902744 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f5b5000f070 con 0x7f5b5c070430 2023-11-15T04:54:35.460 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.442+0000 7f5b4affd700 1 -- 172.21.15.163:0/2900902744 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5b50025b70 con 0x7f5b5c070430 2023-11-15T04:54:35.460 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.443+0000 7f5b60a05700 1 -- 172.21.15.163:0/2900902744 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b5c045ad0 con 0x7f5b5c070430 2023-11-15T04:54:35.461 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.446+0000 7f5b4affd700 1 -- 172.21.15.163:0/2900902744 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5b50016070 con 0x7f5b5c070430 2023-11-15T04:54:35.511 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.510+0000 7f5b60a05700 1 -- 172.21.15.163:0/2900902744 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f5b5c1047f0 con 0x7f5b5c070430 2023-11-15T04:54:35.512 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.510+0000 7f5b4affd700 1 -- 172.21.15.163:0/2900902744 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f5b50006a70 con 0x7f5b5c070430 2023-11-15T04:54:35.513 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.511+0000 7f5b60a05700 1 -- 172.21.15.163:0/2900902744 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b5c070430 msgr2=0x7f5b5c0fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:35.513 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.511+0000 7f5b60a05700 1 --2- 172.21.15.163:0/2900902744 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b5c070430 0x7f5b5c0fc360 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5b5000b190 tx=0x7f5b50006ee0).stop 2023-11-15T04:54:35.513 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.511+0000 7f5b60a05700 1 -- 172.21.15.163:0/2900902744 shutdown_connections 2023-11-15T04:54:35.513 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.511+0000 7f5b60a05700 1 --2- 172.21.15.163:0/2900902744 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b5c070430 0x7f5b5c0fc360 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:35.513 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.511+0000 7f5b60a05700 1 -- 172.21.15.163:0/2900902744 >> 172.21.15.163:0/2900902744 conn(0x7f5b5c0f63c0 msgr2=0x7f5b5c0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:35.514 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.511+0000 7f5b60a05700 1 -- 172.21.15.163:0/2900902744 shutdown_connections 2023-11-15T04:54:35.514 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:35.511+0000 7f5b60a05700 1 -- 172.21.15.163:0/2900902744 wait complete. 2023-11-15T04:54:35.840 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[111936]: audit 2023-11-15T04:54:34.712368+0000 mon.smithi163 (mon.0) 8 : audit [INF] from='client.? 172.21.15.163:0/1503331825' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-15T04:54:35.840 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[111936]: audit 2023-11-15T04:54:34.715489+0000 mon.smithi163 (mon.0) 9 : audit [INF] from='client.? 172.21.15.163:0/1503331825' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-15T04:54:35.873 INFO:teuthology.orchestra.run.smithi163.stderr:Restarting the monitor... 2023-11-15T04:54:35.874 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl restart ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mon.smithi163 2023-11-15T04:54:36.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:35 smithi163 systemd[1]: Stopping Ceph mon.smithi163 for 01689f7e-8373-11ee-8dbe-212e2dc638e7... 2023-11-15T04:54:36.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[111936]: debug 2023-11-15T04:54:35.943+0000 7f76994c1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi163 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-15T04:54:36.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[111936]: debug 2023-11-15T04:54:35.943+0000 7f76994c1700 -1 mon.smithi163@0(leader) e1 *** Got Signal Terminated *** 2023-11-15T04:54:36.646 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 podman[112261]: ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163 2023-11-15T04:54:36.647 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 systemd[1]: ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mon.smithi163.service: Succeeded. 2023-11-15T04:54:36.647 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 systemd[1]: Stopped Ceph mon.smithi163 for 01689f7e-8373-11ee-8dbe-212e2dc638e7. 2023-11-15T04:54:36.647 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 systemd[1]: Starting Ceph mon.smithi163 for 01689f7e-8373-11ee-8dbe-212e2dc638e7... 2023-11-15T04:54:36.714 INFO:teuthology.orchestra.run.smithi163.stderr:Setting mon public_network to 172.21.0.0/20 2023-11-15T04:54:36.715 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2023-11-15T04:54:37.081 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.076+0000 7efeaca1e700 1 Processor -- start 2023-11-15T04:54:37.081 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.076+0000 7efeaca1e700 1 -- start start 2023-11-15T04:54:37.082 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.076+0000 7efeaca1e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efea8070430 0x7efea80fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:37.082 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.076+0000 7efeaca1e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea80fedd0 con 0x7efea8070430 2023-11-15T04:54:37.082 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.077+0000 7efea659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efea8070430 0x7efea80fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:37.082 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.077+0000 7efea659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efea8070430 0x7efea80fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40166/0 (socket says 172.21.15.163:40166) 2023-11-15T04:54:37.083 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.077+0000 7efea659c700 1 -- 172.21.15.163:0/2030345331 learned_addr learned my addr 172.21.15.163:0/2030345331 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:37.083 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.077+0000 7efea659c700 1 -- 172.21.15.163:0/2030345331 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea80fef10 con 0x7efea8070430 2023-11-15T04:54:37.083 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.077+0000 7efea659c700 1 --2- 172.21.15.163:0/2030345331 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efea8070430 0x7efea80fe8e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7efe9c00b5f0 tx=0x7efe9c006ee0).ready entity=mon.0 client_cookie=861d66d6d021d29d server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:37.083 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.078+0000 7efea559a700 1 -- 172.21.15.163:0/2030345331 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe9c00d600 con 0x7efea8070430 2023-11-15T04:54:37.083 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.078+0000 7efea559a700 1 -- 172.21.15.163:0/2030345331 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7efe9c00dbd0 con 0x7efea8070430 2023-11-15T04:54:37.084 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.078+0000 7efea559a700 1 -- 172.21.15.163:0/2030345331 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe9c011c00 con 0x7efea8070430 2023-11-15T04:54:37.084 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.078+0000 7efeaca1e700 1 -- 172.21.15.163:0/2030345331 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efea8070430 msgr2=0x7efea80fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:37.084 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.078+0000 7efeaca1e700 1 --2- 172.21.15.163:0/2030345331 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efea8070430 0x7efea80fe8e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7efe9c00b5f0 tx=0x7efe9c006ee0).stop 2023-11-15T04:54:37.084 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.078+0000 7efeaca1e700 1 -- 172.21.15.163:0/2030345331 shutdown_connections 2023-11-15T04:54:37.084 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.078+0000 7efeaca1e700 1 --2- 172.21.15.163:0/2030345331 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efea8070430 0x7efea80fe8e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:37.085 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.078+0000 7efeaca1e700 1 -- 172.21.15.163:0/2030345331 >> 172.21.15.163:0/2030345331 conn(0x7efea80f63c0 msgr2=0x7efea80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:37.085 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.078+0000 7efeaca1e700 1 -- 172.21.15.163:0/2030345331 shutdown_connections 2023-11-15T04:54:37.085 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.078+0000 7efeaca1e700 1 -- 172.21.15.163:0/2030345331 wait complete. 2023-11-15T04:54:37.085 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.079+0000 7efeaca1e700 1 Processor -- start 2023-11-15T04:54:37.085 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.079+0000 7efeaca1e700 1 -- start start 2023-11-15T04:54:37.086 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.079+0000 7efeaca1e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efea8070430 0x7efea80fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:37.086 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.079+0000 7efeaca1e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea80fa7e0 con 0x7efea8070430 2023-11-15T04:54:37.086 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.079+0000 7efea659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efea8070430 0x7efea80fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:37.086 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.079+0000 7efea659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efea8070430 0x7efea80fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40176/0 (socket says 172.21.15.163:40176) 2023-11-15T04:54:37.086 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.079+0000 7efea659c700 1 -- 172.21.15.163:0/3668329711 learned_addr learned my addr 172.21.15.163:0/3668329711 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:37.087 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.079+0000 7efea659c700 1 -- 172.21.15.163:0/3668329711 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c00b200 con 0x7efea8070430 2023-11-15T04:54:37.087 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.080+0000 7efea659c700 1 --2- 172.21.15.163:0/3668329711 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efea8070430 0x7efea80fc360 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7efe9c002f40 tx=0x7efe9c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:37.088 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.080+0000 7efe96ffd700 1 -- 172.21.15.163:0/3668329711 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe9c011690 con 0x7efea8070430 2023-11-15T04:54:37.088 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.080+0000 7efe96ffd700 1 -- 172.21.15.163:0/3668329711 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7efe9c01b880 con 0x7efea8070430 2023-11-15T04:54:37.088 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.080+0000 7efeaca1e700 1 -- 172.21.15.163:0/3668329711 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea80fa980 con 0x7efea8070430 2023-11-15T04:54:37.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.080+0000 7efe96ffd700 1 -- 172.21.15.163:0/3668329711 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe9c01a6b0 con 0x7efea8070430 2023-11-15T04:54:37.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.080+0000 7efeaca1e700 1 -- 172.21.15.163:0/3668329711 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea80fad90 con 0x7efea8070430 2023-11-15T04:54:37.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.080+0000 7efe96ffd700 1 -- 172.21.15.163:0/3668329711 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7efe9c00f070 con 0x7efea8070430 2023-11-15T04:54:37.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.080+0000 7efe96ffd700 1 -- 172.21.15.163:0/3668329711 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7efe9c0259f0 con 0x7efea8070430 2023-11-15T04:54:37.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.081+0000 7efeaca1e700 1 -- 172.21.15.163:0/3668329711 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea8045ad0 con 0x7efea8070430 2023-11-15T04:54:37.090 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.083+0000 7efe96ffd700 1 -- 172.21.15.163:0/3668329711 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7efe9c016070 con 0x7efea8070430 2023-11-15T04:54:37.148 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.146+0000 7efeaca1e700 1 -- 172.21.15.163:0/3668329711 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7efea80ffc80 con 0x7efea8070430 2023-11-15T04:54:37.150 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.149+0000 7efe96ffd700 1 -- 172.21.15.163:0/3668329711 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7efe9c01bd60 con 0x7efea8070430 2023-11-15T04:54:37.151 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.149+0000 7efeaca1e700 1 -- 172.21.15.163:0/3668329711 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efea8070430 msgr2=0x7efea80fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:37.151 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.149+0000 7efeaca1e700 1 --2- 172.21.15.163:0/3668329711 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efea8070430 0x7efea80fc360 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7efe9c002f40 tx=0x7efe9c006ee0).stop 2023-11-15T04:54:37.151 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.150+0000 7efeaca1e700 1 -- 172.21.15.163:0/3668329711 shutdown_connections 2023-11-15T04:54:37.152 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.150+0000 7efeaca1e700 1 --2- 172.21.15.163:0/3668329711 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efea8070430 0x7efea80fc360 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:37.152 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.150+0000 7efeaca1e700 1 -- 172.21.15.163:0/3668329711 >> 172.21.15.163:0/3668329711 conn(0x7efea80f63c0 msgr2=0x7efea80f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:37.152 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.150+0000 7efeaca1e700 1 -- 172.21.15.163:0/3668329711 shutdown_connections 2023-11-15T04:54:37.152 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:37.150+0000 7efeaca1e700 1 -- 172.21.15.163:0/3668329711 wait complete. 2023-11-15T04:54:37.172 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.704+0000 7f60ef348700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-15T04:54:37.172 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.704+0000 7f60ef348700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2023-11-15T04:54:37.172 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.704+0000 7f60ef348700 0 pidfile_write: ignore empty --pid-file 2023-11-15T04:54:37.173 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.708+0000 7f60ef348700 0 load: jerasure load: lrc load: isa 2023-11-15T04:54:37.173 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: RocksDB version: 6.8.1 2023-11-15T04:54:37.173 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.173 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-15T04:54:37.174 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Compile date May 13 2021 2023-11-15T04:54:37.174 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: DB SUMMARY 2023-11-15T04:54:37.174 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.174 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: CURRENT file: CURRENT 2023-11-15T04:54:37.174 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.175 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: IDENTITY file: IDENTITY 2023-11-15T04:54:37.175 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.175 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-11-15T04:54:37.175 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.175 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi163/store.db dir, Total Num: 1, files: 000004.sst 2023-11-15T04:54:37.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 bash[112331]: 0015f169c26dcfa933814c37413e9f09cc4820327620a7a1cf64c4441a742b11 2023-11-15T04:54:37.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi163/store.db: 000006.log size: 92938 ; 2023-11-15T04:54:37.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.176 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.error_if_exists: 0 2023-11-15T04:54:37.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.create_if_missing: 0 2023-11-15T04:54:37.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.paranoid_checks: 1 2023-11-15T04:54:37.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.env: 0x562fe7f2f1c0 2023-11-15T04:54:37.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.fs: Posix File System 2023-11-15T04:54:37.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.info_log: 0x562fe8b19960 2023-11-15T04:54:37.177 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-15T04:54:37.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.statistics: (nil) 2023-11-15T04:54:37.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.use_fsync: 0 2023-11-15T04:54:37.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_log_file_size: 0 2023-11-15T04:54:37.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-15T04:54:37.178 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-15T04:54:37.179 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-15T04:54:37.179 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-15T04:54:37.179 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.allow_fallocate: 1 2023-11-15T04:54:37.179 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-15T04:54:37.180 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-15T04:54:37.180 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.use_direct_reads: 0 2023-11-15T04:54:37.180 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-15T04:54:37.180 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-15T04:54:37.180 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.db_log_dir: 2023-11-15T04:54:37.181 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi163/store.db 2023-11-15T04:54:37.181 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-15T04:54:37.181 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_subcompactions: 1 2023-11-15T04:54:37.181 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_background_flushes: -1 2023-11-15T04:54:37.181 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-15T04:54:37.182 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-15T04:54:37.182 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-15T04:54:37.182 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-15T04:54:37.182 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-15T04:54:37.182 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-15T04:54:37.183 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-15T04:54:37.183 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.write_buffer_manager: 0x562fe8b1ffb0 2023-11-15T04:54:37.183 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-15T04:54:37.183 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-15T04:54:37.183 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-15T04:54:37.183 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-15T04:54:37.184 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-15T04:54:37.184 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-15T04:54:37.184 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-15T04:54:37.184 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-15T04:54:37.184 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-15T04:54:37.185 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.unordered_write: 0 2023-11-15T04:54:37.185 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-15T04:54:37.185 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-15T04:54:37.185 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-15T04:54:37.185 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-15T04:54:37.185 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.row_cache: None 2023-11-15T04:54:37.186 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.wal_filter: None 2023-11-15T04:54:37.186 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-15T04:54:37.186 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-15T04:54:37.186 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.preserve_deletes: 0 2023-11-15T04:54:37.187 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.two_write_queues: 0 2023-11-15T04:54:37.187 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-15T04:54:37.187 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.atomic_flush: 0 2023-11-15T04:54:37.187 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-15T04:54:37.187 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-15T04:54:37.188 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-15T04:54:37.188 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.log_readahead_size: 0 2023-11-15T04:54:37.188 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-15T04:54:37.188 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_background_jobs: 2 2023-11-15T04:54:37.188 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_background_compactions: -1 2023-11-15T04:54:37.189 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-15T04:54:37.189 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-15T04:54:37.189 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-15T04:54:37.191 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-15T04:54:37.191 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-15T04:54:37.191 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-15T04:54:37.191 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-15T04:54:37.192 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-15T04:54:37.192 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_open_files: -1 2023-11-15T04:54:37.192 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-15T04:54:37.192 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-15T04:54:37.193 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-15T04:54:37.193 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-15T04:54:37.193 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Compression algorithms supported: 2023-11-15T04:54:37.193 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-15T04:54:37.194 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: kZSTD supported: 0 2023-11-15T04:54:37.194 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: kXpressCompression supported: 0 2023-11-15T04:54:37.194 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-15T04:54:37.194 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: kLZ4Compression supported: 1 2023-11-15T04:54:37.194 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: kBZip2Compression supported: 0 2023-11-15T04:54:37.195 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: kZlibCompression supported: 1 2023-11-15T04:54:37.195 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: kSnappyCompression supported: 1 2023-11-15T04:54:37.195 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-15T04:54:37.197 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi163/store.db/MANIFEST-000005 2023-11-15T04:54:37.197 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.197 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-15T04:54:37.197 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.197 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-15T04:54:37.198 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.merge_operator: 2023-11-15T04:54:37.198 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.compaction_filter: None 2023-11-15T04:54:37.198 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-15T04:54:37.198 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-15T04:54:37.199 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-15T04:54:37.199 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562fe8a30128) 2023-11-15T04:54:37.199 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cache_index_and_filter_blocks: 1 2023-11-15T04:54:37.199 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-15T04:54:37.199 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-15T04:54:37.200 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: pin_top_level_index_and_filter: 1 2023-11-15T04:54:37.200 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: index_type: 0 2023-11-15T04:54:37.200 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: data_block_index_type: 0 2023-11-15T04:54:37.200 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: index_shortening: 1 2023-11-15T04:54:37.200 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: data_block_hash_table_util_ratio: 0.750000 2023-11-15T04:54:37.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: hash_index_allow_collision: 1 2023-11-15T04:54:37.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: checksum: 1 2023-11-15T04:54:37.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: no_block_cache: 0 2023-11-15T04:54:37.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: block_cache: 0x562fe8a66f10 2023-11-15T04:54:37.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: block_cache_name: BinnedLRUCache 2023-11-15T04:54:37.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: block_cache_options: 2023-11-15T04:54:37.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: capacity : 536870912 2023-11-15T04:54:37.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: num_shard_bits : 4 2023-11-15T04:54:37.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: strict_capacity_limit : 0 2023-11-15T04:54:37.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: high_pri_pool_ratio: 0.000 2023-11-15T04:54:37.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: block_cache_compressed: (nil) 2023-11-15T04:54:37.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: persistent_cache: (nil) 2023-11-15T04:54:37.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: block_size: 4096 2023-11-15T04:54:37.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: block_size_deviation: 10 2023-11-15T04:54:37.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: block_restart_interval: 16 2023-11-15T04:54:37.204 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: index_block_restart_interval: 1 2023-11-15T04:54:37.204 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: metadata_block_size: 4096 2023-11-15T04:54:37.204 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: partition_filters: 0 2023-11-15T04:54:37.204 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: use_delta_encoding: 1 2023-11-15T04:54:37.204 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-15T04:54:37.205 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: whole_key_filtering: 1 2023-11-15T04:54:37.205 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: verify_compression: 0 2023-11-15T04:54:37.205 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: read_amp_bytes_per_bit: 0 2023-11-15T04:54:37.205 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: format_version: 2 2023-11-15T04:54:37.205 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: enable_index_compression: 1 2023-11-15T04:54:37.205 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: block_align: 0 2023-11-15T04:54:37.206 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.206 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-15T04:54:37.206 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-15T04:54:37.206 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.compression: NoCompression 2023-11-15T04:54:37.207 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-15T04:54:37.207 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-15T04:54:37.207 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-15T04:54:37.207 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.num_levels: 7 2023-11-15T04:54:37.208 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-15T04:54:37.208 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-15T04:54:37.208 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-15T04:54:37.208 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-15T04:54:37.209 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-15T04:54:37.209 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-15T04:54:37.209 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-15T04:54:37.210 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-15T04:54:37.210 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-15T04:54:37.210 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-15T04:54:37.210 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-15T04:54:37.211 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-15T04:54:37.211 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-15T04:54:37.211 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-15T04:54:37.211 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-15T04:54:37.211 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-15T04:54:37.212 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-15T04:54:37.212 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-15T04:54:37.212 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-15T04:54:37.212 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-15T04:54:37.213 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-15T04:54:37.213 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-15T04:54:37.213 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-15T04:54:37.213 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-15T04:54:37.213 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-15T04:54:37.213 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-15T04:54:37.214 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-15T04:54:37.214 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-15T04:54:37.214 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-15T04:54:37.214 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-15T04:54:37.214 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-15T04:54:37.215 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-15T04:54:37.215 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-15T04:54:37.215 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-15T04:54:37.215 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-15T04:54:37.215 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-15T04:54:37.216 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-15T04:54:37.216 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-15T04:54:37.216 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-15T04:54:37.216 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-15T04:54:37.216 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-15T04:54:37.216 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-15T04:54:37.217 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-15T04:54:37.217 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-15T04:54:37.217 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-15T04:54:37.217 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-15T04:54:37.217 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-15T04:54:37.218 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.table_properties_collectors: 2023-11-15T04:54:37.218 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.inplace_update_support: 0 2023-11-15T04:54:37.218 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-15T04:54:37.218 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-15T04:54:37.218 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-15T04:54:37.218 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-15T04:54:37.219 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.bloom_locality: 0 2023-11-15T04:54:37.219 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.max_successive_merges: 0 2023-11-15T04:54:37.219 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-15T04:54:37.219 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-15T04:54:37.220 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-15T04:54:37.220 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-15T04:54:37.220 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.ttl: 2592000 2023-11-15T04:54:37.220 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-15T04:54:37.220 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi163/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-15T04:54:37.221 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.221 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.709+0000 7f60ef348700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-11-15T04:54:37.221 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.221 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.710+0000 7f60ef348700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700024076711415, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-11-15T04:54:37.221 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.710+0000 7f60ef348700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-11-15T04:54:37.222 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.710+0000 7f60ef348700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-11-15T04:54:37.222 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.710+0000 7f60ef348700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700024076712262, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88718, "table_properties": {"data_size": 86833, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79974, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700024076, "oldest_key_time": 3, "file_creation_time": 0}} 2023-11-15T04:54:37.222 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.710+0000 7f60ef348700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-11-15T04:54:37.222 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.223 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 systemd[1]: Started Ceph mon.smithi163 for 01689f7e-8373-11ee-8dbe-212e2dc638e7. 2023-11-15T04:54:37.223 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.711+0000 7f60ef348700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700024076713363, "job": 1, "event": "recovery_finished"} 2023-11-15T04:54:37.223 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.712+0000 7f60ef348700 4 rocksdb: DB pointer 0x562fe8b2b800 2023-11-15T04:54:37.223 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.713+0000 7f60d6160700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-15T04:54:37.223 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.713+0000 7f60d6160700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-15T04:54:37.224 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: ** DB Stats ** 2023-11-15T04:54:37.224 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T04:54:37.224 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-15T04:54:37.224 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-15T04:54:37.224 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T04:54:37.225 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-15T04:54:37.225 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-15T04:54:37.225 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T04:54:37.225 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.225 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: ** Compaction Stats [default] ** 2023-11-15T04:54:37.226 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:54:37.226 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:54:37.226 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: L0 2/0 88.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.3 0.00 0.00 1 0.001 0 0 2023-11-15T04:54:37.226 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Sum 2/0 88.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.3 0.00 0.00 1 0.001 0 0 2023-11-15T04:54:37.226 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.3 0.00 0.00 1 0.001 0 0 2023-11-15T04:54:37.227 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.227 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: ** Compaction Stats [default] ** 2023-11-15T04:54:37.227 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:54:37.227 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:54:37.227 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 150.3 0.00 0.00 1 0.001 0 0 2023-11-15T04:54:37.227 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T04:54:37.228 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T04:54:37.228 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T04:54:37.228 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T04:54:37.228 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T04:54:37.228 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: AddFile(Keys): cumulative 0, interval 0 2023-11-15T04:54:37.228 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Cumulative compaction: 0.00 GB write, 22.30 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:54:37.229 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Interval compaction: 0.00 GB write, 22.30 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:54:37.229 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T04:54:37.229 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.229 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: ** File Read Latency Histogram By Level [default] ** 2023-11-15T04:54:37.229 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.230 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: ** Compaction Stats [default] ** 2023-11-15T04:54:37.230 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:54:37.230 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:54:37.230 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: L0 2/0 88.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.3 0.00 0.00 1 0.001 0 0 2023-11-15T04:54:37.231 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Sum 2/0 88.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.3 0.00 0.00 1 0.001 0 0 2023-11-15T04:54:37.231 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T04:54:37.231 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.231 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: ** Compaction Stats [default] ** 2023-11-15T04:54:37.231 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T04:54:37.232 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T04:54:37.232 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 150.3 0.00 0.00 1 0.001 0 0 2023-11-15T04:54:37.232 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T04:54:37.232 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T04:54:37.232 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T04:54:37.233 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T04:54:37.233 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T04:54:37.233 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: AddFile(Keys): cumulative 0, interval 0 2023-11-15T04:54:37.233 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Cumulative compaction: 0.00 GB write, 22.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:54:37.233 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T04:54:37.234 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T04:54:37.234 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.234 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: ** File Read Latency Histogram By Level [default] ** 2023-11-15T04:54:37.234 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.234 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.713+0000 7f60ef348700 0 starting mon.smithi163 rank 0 at public addrs [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] at bind addrs [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi163 fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 2023-11-15T04:54:37.235 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.714+0000 7f60ef348700 1 mon.smithi163@-1(???) e1 preinit fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 2023-11-15T04:54:37.235 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.714+0000 7f60ef348700 0 mon.smithi163@-1(???).mds e1 new map 2023-11-15T04:54:37.235 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.714+0000 7f60ef348700 0 mon.smithi163@-1(???).mds e1 print_map 2023-11-15T04:54:37.235 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: e1 2023-11-15T04:54:37.235 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: enable_multiple, ever_enabled_multiple: 1,1 2023-11-15T04:54:37.236 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-15T04:54:37.236 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: legacy client fscid: -1 2023-11-15T04:54:37.236 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.236 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: No filesystems configured 2023-11-15T04:54:37.236 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: 2023-11-15T04:54:37.237 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.714+0000 7f60ef348700 0 mon.smithi163@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-15T04:54:37.237 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.714+0000 7f60ef348700 0 mon.smithi163@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-15T04:54:37.237 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.714+0000 7f60ef348700 0 mon.smithi163@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-15T04:54:37.237 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.714+0000 7f60ef348700 0 mon.smithi163@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-15T04:54:37.237 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:34.712368+0000 mon.smithi163 (mon.0) 8 : audit [INF] from='client.? 172.21.15.163:0/1503331825' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-15T04:54:37.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:34.715489+0000 mon.smithi163 (mon.0) 9 : audit [INF] from='client.? 172.21.15.163:0/1503331825' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-15T04:54:37.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.714+0000 7f60ef348700 1 mon.smithi163@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-11-15T04:54:37.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.715+0000 7f60ef348700 20 expand_channel_meta expand map: {default=false} 2023-11-15T04:54:37.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.715+0000 7f60ef348700 20 expand_channel_meta from 'false' to 'false' 2023-11-15T04:54:37.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.715+0000 7f60ef348700 20 expand_channel_meta expanded map: {default=false} 2023-11-15T04:54:37.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.715+0000 7f60ef348700 20 expand_channel_meta expand map: {default=info} 2023-11-15T04:54:37.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.715+0000 7f60ef348700 20 expand_channel_meta from 'info' to 'info' 2023-11-15T04:54:37.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.715+0000 7f60ef348700 20 expand_channel_meta expanded map: {default=info} 2023-11-15T04:54:37.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.715+0000 7f60ef348700 20 expand_channel_meta expand map: {default=daemon} 2023-11-15T04:54:37.240 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.715+0000 7f60ef348700 20 expand_channel_meta from 'daemon' to 'daemon' 2023-11-15T04:54:37.240 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.715+0000 7f60ef348700 20 expand_channel_meta expanded map: {default=daemon} 2023-11-15T04:54:37.240 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.715+0000 7f60ef348700 20 expand_channel_meta expand map: {default=debug} 2023-11-15T04:54:37.240 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.715+0000 7f60ef348700 20 expand_channel_meta from 'debug' to 'debug' 2023-11-15T04:54:37.240 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T04:54:36.715+0000 7f60ef348700 20 expand_channel_meta expanded map: {default=debug} 2023-11-15T04:54:37.241 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:54:36.732059+0000 mon.smithi163 (mon.0) 1 : cluster [INF] mon.smithi163 is new leader, mons smithi163 in quorum (ranks 0) 2023-11-15T04:54:37.241 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:54:36.732141+0000 mon.smithi163 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi163=[v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0]} 2023-11-15T04:54:37.241 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:54:36.732182+0000 mon.smithi163 (mon.0) 3 : cluster [DBG] fsmap 2023-11-15T04:54:37.241 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:54:36.732211+0000 mon.smithi163 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-15T04:54:37.241 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:54:36.732647+0000 mon.smithi163 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-11-15T04:54:37.538 INFO:teuthology.orchestra.run.smithi163.stderr:Wrote config to /etc/ceph/ceph.conf 2023-11-15T04:54:37.538 INFO:teuthology.orchestra.run.smithi163.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-11-15T04:54:37.539 INFO:teuthology.orchestra.run.smithi163.stderr:Creating mgr... 2023-11-15T04:54:37.539 INFO:teuthology.orchestra.run.smithi163.stderr:Verifying port 9283 ... 2023-11-15T04:54:37.541 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl daemon-reload 2023-11-15T04:54:37.637 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl stop ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mgr.smithi163.qesyrx 2023-11-15T04:54:37.659 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl reset-failed ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mgr.smithi163.qesyrx 2023-11-15T04:54:37.665 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: Failed to reset failed state of unit ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mgr.smithi163.qesyrx.service: Unit ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mgr.smithi163.qesyrx.service not loaded. 2023-11-15T04:54:37.665 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl enable ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mgr.smithi163.qesyrx 2023-11-15T04:54:37.671 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: Created symlink /etc/systemd/system/ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7.target.wants/ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mgr.smithi163.qesyrx.service → /etc/systemd/system/ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@.service. 2023-11-15T04:54:37.758 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl start ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mgr.smithi163.qesyrx 2023-11-15T04:54:38.065 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T04:54:38.072 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: disabled 2023-11-15T04:54:38.073 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T04:54:38.080 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: inactive 2023-11-15T04:54:38.081 INFO:teuthology.orchestra.run.smithi163.stderr:firewalld.service is not enabled 2023-11-15T04:54:38.081 INFO:teuthology.orchestra.run.smithi163.stderr:Not possible to enable service . firewalld.service is not available 2023-11-15T04:54:38.081 INFO:teuthology.orchestra.run.smithi163.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-15T04:54:38.082 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T04:54:38.090 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: disabled 2023-11-15T04:54:38.090 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T04:54:38.096 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: inactive 2023-11-15T04:54:38.097 INFO:teuthology.orchestra.run.smithi163.stderr:firewalld.service is not enabled 2023-11-15T04:54:38.098 INFO:teuthology.orchestra.run.smithi163.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-11-15T04:54:38.098 INFO:teuthology.orchestra.run.smithi163.stderr:Waiting for mgr to start... 2023-11-15T04:54:38.098 INFO:teuthology.orchestra.run.smithi163.stderr:Waiting for mgr... 2023-11-15T04:54:38.099 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T04:54:38.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:38 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:37.150454+0000 mon.smithi163 (mon.0) 6 : audit [INF] from='client.? 172.21.15.163:0/3668329711' entity='client.admin' 2023-11-15T04:54:38.482 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.479+0000 7f185e835700 1 Processor -- start 2023-11-15T04:54:38.484 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.479+0000 7f185e835700 1 -- start start 2023-11-15T04:54:38.484 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.479+0000 7f185e835700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1858070390 0x7f1858100da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:38.484 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.479+0000 7f185e835700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18580fc790 con 0x7f1858070390 2023-11-15T04:54:38.485 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.480+0000 7f1857fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1858070390 0x7f1858100da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:38.485 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.480+0000 7f1857fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1858070390 0x7f1858100da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40194/0 (socket says 172.21.15.163:40194) 2023-11-15T04:54:38.485 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.480+0000 7f1857fff700 1 -- 172.21.15.163:0/2144531882 learned_addr learned my addr 172.21.15.163:0/2144531882 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:38.485 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.480+0000 7f1857fff700 1 -- 172.21.15.163:0/2144531882 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1858101290 con 0x7f1858070390 2023-11-15T04:54:38.485 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.480+0000 7f1857fff700 1 --2- 172.21.15.163:0/2144531882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1858070390 0x7f1858100da0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f184800b5f0 tx=0x7f1848006ee0).ready entity=mon.0 client_cookie=2f01126da2be0faa server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:38.486 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.481+0000 7f1856ffd700 1 -- 172.21.15.163:0/2144531882 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f184800d600 con 0x7f1858070390 2023-11-15T04:54:38.486 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.481+0000 7f1856ffd700 1 -- 172.21.15.163:0/2144531882 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f184800dbd0 con 0x7f1858070390 2023-11-15T04:54:38.486 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.481+0000 7f1856ffd700 1 -- 172.21.15.163:0/2144531882 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1848011c00 con 0x7f1858070390 2023-11-15T04:54:38.486 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.481+0000 7f185e835700 1 -- 172.21.15.163:0/2144531882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1858070390 msgr2=0x7f1858100da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:38.487 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.481+0000 7f185e835700 1 --2- 172.21.15.163:0/2144531882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1858070390 0x7f1858100da0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f184800b5f0 tx=0x7f1848006ee0).stop 2023-11-15T04:54:38.487 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.481+0000 7f185e835700 1 -- 172.21.15.163:0/2144531882 shutdown_connections 2023-11-15T04:54:38.487 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.481+0000 7f185e835700 1 --2- 172.21.15.163:0/2144531882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1858070390 0x7f1858100da0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:38.487 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.481+0000 7f185e835700 1 -- 172.21.15.163:0/2144531882 >> 172.21.15.163:0/2144531882 conn(0x7f18580f63c0 msgr2=0x7f18580f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:38.487 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.481+0000 7f185e835700 1 -- 172.21.15.163:0/2144531882 shutdown_connections 2023-11-15T04:54:38.488 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.481+0000 7f185e835700 1 -- 172.21.15.163:0/2144531882 wait complete. 2023-11-15T04:54:38.488 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.482+0000 7f185e835700 1 Processor -- start 2023-11-15T04:54:38.488 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.482+0000 7f185e835700 1 -- start start 2023-11-15T04:54:38.488 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.482+0000 7f185e835700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1858070390 0x7f185810cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:38.488 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.482+0000 7f185e835700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185810d0f0 con 0x7f1858070390 2023-11-15T04:54:38.490 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.483+0000 7f1857fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1858070390 0x7f185810cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:38.490 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.483+0000 7f1857fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1858070390 0x7f185810cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40200/0 (socket says 172.21.15.163:40200) 2023-11-15T04:54:38.490 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.483+0000 7f1857fff700 1 -- 172.21.15.163:0/691599581 learned_addr learned my addr 172.21.15.163:0/691599581 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:38.491 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.483+0000 7f1857fff700 1 -- 172.21.15.163:0/691599581 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184800b200 con 0x7f1858070390 2023-11-15T04:54:38.491 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.483+0000 7f1857fff700 1 --2- 172.21.15.163:0/691599581 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1858070390 0x7f185810cc00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f184800b190 tx=0x7f1848006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:38.491 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.483+0000 7f1854ff9700 1 -- 172.21.15.163:0/691599581 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f184801b410 con 0x7f1858070390 2023-11-15T04:54:38.491 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.483+0000 7f185e835700 1 -- 172.21.15.163:0/691599581 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f185810d2f0 con 0x7f1858070390 2023-11-15T04:54:38.491 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.483+0000 7f1854ff9700 1 -- 172.21.15.163:0/691599581 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f184801b9e0 con 0x7f1858070390 2023-11-15T04:54:38.492 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.483+0000 7f1854ff9700 1 -- 172.21.15.163:0/691599581 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f184801a880 con 0x7f1858070390 2023-11-15T04:54:38.492 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.483+0000 7f185e835700 1 -- 172.21.15.163:0/691599581 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f185810d700 con 0x7f1858070390 2023-11-15T04:54:38.492 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.483+0000 7f1854ff9700 1 -- 172.21.15.163:0/691599581 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f184800f070 con 0x7f1858070390 2023-11-15T04:54:38.492 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.483+0000 7f1854ff9700 1 -- 172.21.15.163:0/691599581 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1848025b70 con 0x7f1858070390 2023-11-15T04:54:38.492 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.484+0000 7f185e835700 1 -- 172.21.15.163:0/691599581 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1858045ad0 con 0x7f1858070390 2023-11-15T04:54:38.493 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.487+0000 7f1854ff9700 1 -- 172.21.15.163:0/691599581 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f1848016070 con 0x7f1858070390 2023-11-15T04:54:38.566 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.556+0000 7f185e835700 1 -- 172.21.15.163:0/691599581 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1858106ac0 con 0x7f1858070390 2023-11-15T04:54:38.566 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.556+0000 7f1854ff9700 1 -- 172.21.15.163:0/691599581 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f1848006a70 con 0x7f1858070390 2023-11-15T04:54:38.567 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.567 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: { 2023-11-15T04:54:38.567 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "fsid": "01689f7e-8373-11ee-8dbe-212e2dc638e7", 2023-11-15T04:54:38.567 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "health": { 2023-11-15T04:54:38.568 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T04:54:38.568 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T04:54:38.568 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T04:54:38.568 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2023-11-15T04:54:38.568 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T04:54:38.569 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T04:54:38.569 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 0 2023-11-15T04:54:38.569 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: ], 2023-11-15T04:54:38.569 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T04:54:38.570 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "smithi163" 2023-11-15T04:54:38.570 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: ], 2023-11-15T04:54:38.570 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "quorum_age": 1, 2023-11-15T04:54:38.570 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T04:54:38.570 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:54:38.571 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T04:54:38.571 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T04:54:38.571 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2023-11-15T04:54:38.571 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T04:54:38.571 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:54:38.571 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T04:54:38.572 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T04:54:38.572 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T04:54:38.572 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T04:54:38.572 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T04:54:38.572 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T04:54:38.573 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2023-11-15T04:54:38.573 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T04:54:38.573 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T04:54:38.573 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T04:54:38.573 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T04:54:38.573 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T04:54:38.574 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T04:54:38.574 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T04:54:38.574 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T04:54:38.574 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T04:54:38.574 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2023-11-15T04:54:38.575 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T04:54:38.575 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:54:38.575 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T04:54:38.575 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T04:54:38.575 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2023-11-15T04:54:38.576 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T04:54:38.576 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "available": false, 2023-11-15T04:54:38.576 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T04:54:38.576 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T04:54:38.576 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "iostat", 2023-11-15T04:54:38.576 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "restful" 2023-11-15T04:54:38.577 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: ], 2023-11-15T04:54:38.577 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:54:38.577 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2023-11-15T04:54:38.577 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T04:54:38.577 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:54:38.578 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "modified": "2023-11-15T04:54:33.468510+0000", 2023-11-15T04:54:38.578 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:54:38.578 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2023-11-15T04:54:38.578 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T04:54:38.578 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: } 2023-11-15T04:54:38.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.557+0000 7f185e835700 1 -- 172.21.15.163:0/691599581 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1858070390 msgr2=0x7f185810cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:38.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.557+0000 7f185e835700 1 --2- 172.21.15.163:0/691599581 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1858070390 0x7f185810cc00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f184800b190 tx=0x7f1848006ee0).stop 2023-11-15T04:54:38.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.557+0000 7f185e835700 1 -- 172.21.15.163:0/691599581 shutdown_connections 2023-11-15T04:54:38.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.557+0000 7f185e835700 1 --2- 172.21.15.163:0/691599581 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1858070390 0x7f185810cc00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:38.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.557+0000 7f185e835700 1 -- 172.21.15.163:0/691599581 >> 172.21.15.163:0/691599581 conn(0x7f18580f63c0 msgr2=0x7f18580f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:38.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.557+0000 7f185e835700 1 -- 172.21.15.163:0/691599581 shutdown_connections 2023-11-15T04:54:38.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:38.557+0000 7f185e835700 1 -- 172.21.15.163:0/691599581 wait complete. 2023-11-15T04:54:38.954 INFO:teuthology.orchestra.run.smithi163.stderr:mgr not available, waiting (1/15)... 2023-11-15T04:54:39.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:39 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:38.557851+0000 mon.smithi163 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.163:0/691599581' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T04:54:40.957 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T04:54:41.359 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.357+0000 7ff455ab1700 1 Processor -- start 2023-11-15T04:54:41.360 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.357+0000 7ff455ab1700 1 -- start start 2023-11-15T04:54:41.360 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.357+0000 7ff455ab1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4500724b0 0x7ff450070a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:41.360 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.357+0000 7ff455ab1700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4500728d0 con 0x7ff4500724b0 2023-11-15T04:54:41.360 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.357+0000 7ff44f7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4500724b0 0x7ff450070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:41.361 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.357+0000 7ff44f7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4500724b0 0x7ff450070a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36294/0 (socket says 172.21.15.163:36294) 2023-11-15T04:54:41.362 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.358+0000 7ff44f7fe700 1 -- 172.21.15.163:0/3932941044 learned_addr learned my addr 172.21.15.163:0/3932941044 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:41.362 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.358+0000 7ff44f7fe700 1 -- 172.21.15.163:0/3932941044 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff450070f40 con 0x7ff4500724b0 2023-11-15T04:54:41.363 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.358+0000 7ff44f7fe700 1 --2- 172.21.15.163:0/3932941044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4500724b0 0x7ff450070a50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7ff44000b5f0 tx=0x7ff440006ee0).ready entity=mon.0 client_cookie=71e205eb145eb3fd server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:41.363 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.358+0000 7ff44e7fc700 1 -- 172.21.15.163:0/3932941044 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff44000d600 con 0x7ff4500724b0 2023-11-15T04:54:41.363 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.358+0000 7ff44e7fc700 1 -- 172.21.15.163:0/3932941044 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff44000dbd0 con 0x7ff4500724b0 2023-11-15T04:54:41.363 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.358+0000 7ff44e7fc700 1 -- 172.21.15.163:0/3932941044 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff440011c00 con 0x7ff4500724b0 2023-11-15T04:54:41.364 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.359+0000 7ff455ab1700 1 -- 172.21.15.163:0/3932941044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4500724b0 msgr2=0x7ff450070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:41.364 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.359+0000 7ff455ab1700 1 --2- 172.21.15.163:0/3932941044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4500724b0 0x7ff450070a50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7ff44000b5f0 tx=0x7ff440006ee0).stop 2023-11-15T04:54:41.364 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.359+0000 7ff455ab1700 1 -- 172.21.15.163:0/3932941044 shutdown_connections 2023-11-15T04:54:41.364 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.359+0000 7ff455ab1700 1 --2- 172.21.15.163:0/3932941044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4500724b0 0x7ff450070a50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:41.364 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.359+0000 7ff455ab1700 1 -- 172.21.15.163:0/3932941044 >> 172.21.15.163:0/3932941044 conn(0x7ff4500f74c0 msgr2=0x7ff4500f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:41.365 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.359+0000 7ff455ab1700 1 -- 172.21.15.163:0/3932941044 shutdown_connections 2023-11-15T04:54:41.365 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.359+0000 7ff455ab1700 1 -- 172.21.15.163:0/3932941044 wait complete. 2023-11-15T04:54:41.366 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.359+0000 7ff455ab1700 1 Processor -- start 2023-11-15T04:54:41.366 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.359+0000 7ff455ab1700 1 -- start start 2023-11-15T04:54:41.367 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.360+0000 7ff455ab1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4500724b0 0x7ff45010ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:41.367 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.360+0000 7ff455ab1700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45010d1c0 con 0x7ff4500724b0 2023-11-15T04:54:41.367 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.360+0000 7ff44f7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4500724b0 0x7ff45010ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:41.367 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.360+0000 7ff44f7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4500724b0 0x7ff45010ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36296/0 (socket says 172.21.15.163:36296) 2023-11-15T04:54:41.368 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.360+0000 7ff44f7fe700 1 -- 172.21.15.163:0/1405903384 learned_addr learned my addr 172.21.15.163:0/1405903384 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:41.368 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.360+0000 7ff44f7fe700 1 -- 172.21.15.163:0/1405903384 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44000b200 con 0x7ff4500724b0 2023-11-15T04:54:41.368 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.360+0000 7ff44f7fe700 1 --2- 172.21.15.163:0/1405903384 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4500724b0 0x7ff45010ccd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ff44000b190 tx=0x7ff440006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:41.368 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.360+0000 7ff437fff700 1 -- 172.21.15.163:0/1405903384 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff44001b410 con 0x7ff4500724b0 2023-11-15T04:54:41.369 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.360+0000 7ff437fff700 1 -- 172.21.15.163:0/1405903384 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff44001b9e0 con 0x7ff4500724b0 2023-11-15T04:54:41.369 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.360+0000 7ff455ab1700 1 -- 172.21.15.163:0/1405903384 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45010d3c0 con 0x7ff4500724b0 2023-11-15T04:54:41.369 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.360+0000 7ff437fff700 1 -- 172.21.15.163:0/1405903384 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff44001a880 con 0x7ff4500724b0 2023-11-15T04:54:41.369 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.360+0000 7ff455ab1700 1 -- 172.21.15.163:0/1405903384 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff45010d7d0 con 0x7ff4500724b0 2023-11-15T04:54:41.369 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.361+0000 7ff437fff700 1 -- 172.21.15.163:0/1405903384 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7ff44000f070 con 0x7ff4500724b0 2023-11-15T04:54:41.370 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.361+0000 7ff437fff700 1 -- 172.21.15.163:0/1405903384 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ff440025b40 con 0x7ff4500724b0 2023-11-15T04:54:41.370 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.361+0000 7ff455ab1700 1 -- 172.21.15.163:0/1405903384 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff450045ad0 con 0x7ff4500724b0 2023-11-15T04:54:41.370 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.364+0000 7ff437fff700 1 -- 172.21.15.163:0/1405903384 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7ff440016070 con 0x7ff4500724b0 2023-11-15T04:54:41.443 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.432+0000 7ff455ab1700 1 -- 172.21.15.163:0/1405903384 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff45010e6d0 con 0x7ff4500724b0 2023-11-15T04:54:41.443 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.432+0000 7ff437fff700 1 -- 172.21.15.163:0/1405903384 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7ff440025d80 con 0x7ff4500724b0 2023-11-15T04:54:41.443 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.443 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: { 2023-11-15T04:54:41.444 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "fsid": "01689f7e-8373-11ee-8dbe-212e2dc638e7", 2023-11-15T04:54:41.444 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "health": { 2023-11-15T04:54:41.444 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T04:54:41.444 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T04:54:41.444 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T04:54:41.445 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2023-11-15T04:54:41.445 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T04:54:41.445 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T04:54:41.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 0 2023-11-15T04:54:41.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: ], 2023-11-15T04:54:41.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T04:54:41.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "smithi163" 2023-11-15T04:54:41.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: ], 2023-11-15T04:54:41.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "quorum_age": 4, 2023-11-15T04:54:41.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T04:54:41.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:54:41.448 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T04:54:41.448 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T04:54:41.448 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2023-11-15T04:54:41.448 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T04:54:41.449 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:54:41.449 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T04:54:41.449 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T04:54:41.449 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T04:54:41.449 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T04:54:41.450 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T04:54:41.450 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T04:54:41.450 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2023-11-15T04:54:41.450 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T04:54:41.451 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T04:54:41.451 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T04:54:41.451 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T04:54:41.451 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T04:54:41.451 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T04:54:41.452 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T04:54:41.452 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T04:54:41.452 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T04:54:41.452 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2023-11-15T04:54:41.452 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T04:54:41.452 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:54:41.453 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T04:54:41.453 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T04:54:41.453 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2023-11-15T04:54:41.453 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T04:54:41.454 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "available": false, 2023-11-15T04:54:41.454 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T04:54:41.454 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T04:54:41.454 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "iostat", 2023-11-15T04:54:41.454 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "restful" 2023-11-15T04:54:41.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: ], 2023-11-15T04:54:41.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:54:41.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2023-11-15T04:54:41.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T04:54:41.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:54:41.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "modified": "2023-11-15T04:54:33.468510+0000", 2023-11-15T04:54:41.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:54:41.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2023-11-15T04:54:41.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T04:54:41.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: } 2023-11-15T04:54:41.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.433+0000 7ff455ab1700 1 -- 172.21.15.163:0/1405903384 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4500724b0 msgr2=0x7ff45010ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:41.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.433+0000 7ff455ab1700 1 --2- 172.21.15.163:0/1405903384 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4500724b0 0x7ff45010ccd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ff44000b190 tx=0x7ff440006ee0).stop 2023-11-15T04:54:41.458 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.433+0000 7ff455ab1700 1 -- 172.21.15.163:0/1405903384 shutdown_connections 2023-11-15T04:54:41.458 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.433+0000 7ff455ab1700 1 --2- 172.21.15.163:0/1405903384 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4500724b0 0x7ff45010ccd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:41.458 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.433+0000 7ff455ab1700 1 -- 172.21.15.163:0/1405903384 >> 172.21.15.163:0/1405903384 conn(0x7ff4500f74c0 msgr2=0x7ff4500f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:41.459 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.433+0000 7ff455ab1700 1 -- 172.21.15.163:0/1405903384 shutdown_connections 2023-11-15T04:54:41.459 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:41.433+0000 7ff455ab1700 1 -- 172.21.15.163:0/1405903384 wait complete. 2023-11-15T04:54:41.747 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:41 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:41.433724+0000 mon.smithi163 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.163:0/1405903384' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T04:54:41.779 INFO:teuthology.orchestra.run.smithi163.stderr:mgr not available, waiting (2/15)... 2023-11-15T04:54:43.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:43 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:54:43.044263+0000 mon.smithi163 (mon.0) 9 : cluster [INF] Activating manager daemon smithi163.qesyrx 2023-11-15T04:54:43.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:43 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:54:43.046870+0000 mon.smithi163 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi163.qesyrx(active, starting, since 0.0027134s) 2023-11-15T04:54:43.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:43 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:43.047526+0000 mon.smithi163 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.163:0/2203065056' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T04:54:43.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:43 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:43.047724+0000 mon.smithi163 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.163:0/2203065056' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T04:54:43.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:43 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:43.047872+0000 mon.smithi163 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.163:0/2203065056' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T04:54:43.454 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:43 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:43.047980+0000 mon.smithi163 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.163:0/2203065056' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2023-11-15T04:54:43.454 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:43 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:43.048283+0000 mon.smithi163 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.163:0/2203065056' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "mgr metadata", "who": "smithi163.qesyrx", "id": "smithi163.qesyrx"}]: dispatch 2023-11-15T04:54:43.454 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:43 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:54:43.061905+0000 mon.smithi163 (mon.0) 16 : cluster [INF] Manager daemon smithi163.qesyrx is now available 2023-11-15T04:54:43.454 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:43 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:43.069330+0000 mon.smithi163 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.163:0/2203065056' entity='mgr.smithi163.qesyrx' 2023-11-15T04:54:43.455 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:43 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:43.072871+0000 mon.smithi163 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.163:0/2203065056' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:54:43.455 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:43 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:43.073957+0000 mon.smithi163 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.163:0/2203065056' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/trash_purge_schedule"}]: dispatch 2023-11-15T04:54:43.455 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:43 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:43.078359+0000 mon.smithi163 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.163:0/2203065056' entity='mgr.smithi163.qesyrx' 2023-11-15T04:54:43.455 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:43 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:43.080249+0000 mon.smithi163 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.163:0/2203065056' entity='mgr.smithi163.qesyrx' 2023-11-15T04:54:43.781 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T04:54:44.136 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.132+0000 7f3a52375700 1 Processor -- start 2023-11-15T04:54:44.136 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.132+0000 7f3a52375700 1 -- start start 2023-11-15T04:54:44.136 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.132+0000 7f3a52375700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a4c0fab80 0x7f3a4c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:44.136 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.132+0000 7f3a52375700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c0fb490 con 0x7f3a4c0fab80 2023-11-15T04:54:44.137 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.132+0000 7f3a4bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a4c0fab80 0x7f3a4c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:44.137 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.132+0000 7f3a4bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a4c0fab80 0x7f3a4c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36332/0 (socket says 172.21.15.163:36332) 2023-11-15T04:54:44.137 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.132+0000 7f3a4bfff700 1 -- 172.21.15.163:0/2168998806 learned_addr learned my addr 172.21.15.163:0/2168998806 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:44.137 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.132+0000 7f3a4bfff700 1 -- 172.21.15.163:0/2168998806 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a4c0fb5d0 con 0x7f3a4c0fab80 2023-11-15T04:54:44.137 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.133+0000 7f3a4bfff700 1 --2- 172.21.15.163:0/2168998806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a4c0fab80 0x7f3a4c0fafa0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f3a3c00b5f0 tx=0x7f3a3c006ee0).ready entity=mon.0 client_cookie=881f5454e13836be server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:44.138 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.133+0000 7f3a4affd700 1 -- 172.21.15.163:0/2168998806 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a3c00d600 con 0x7f3a4c0fab80 2023-11-15T04:54:44.138 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.133+0000 7f3a4affd700 1 -- 172.21.15.163:0/2168998806 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3a3c00dbd0 con 0x7f3a4c0fab80 2023-11-15T04:54:44.138 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.133+0000 7f3a4affd700 1 -- 172.21.15.163:0/2168998806 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a3c011c00 con 0x7f3a4c0fab80 2023-11-15T04:54:44.138 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.133+0000 7f3a52375700 1 -- 172.21.15.163:0/2168998806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a4c0fab80 msgr2=0x7f3a4c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:44.138 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.133+0000 7f3a52375700 1 --2- 172.21.15.163:0/2168998806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a4c0fab80 0x7f3a4c0fafa0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f3a3c00b5f0 tx=0x7f3a3c006ee0).stop 2023-11-15T04:54:44.139 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.133+0000 7f3a52375700 1 -- 172.21.15.163:0/2168998806 shutdown_connections 2023-11-15T04:54:44.139 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.133+0000 7f3a52375700 1 --2- 172.21.15.163:0/2168998806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a4c0fab80 0x7f3a4c0fafa0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:44.139 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.133+0000 7f3a52375700 1 -- 172.21.15.163:0/2168998806 >> 172.21.15.163:0/2168998806 conn(0x7f3a4c0f63c0 msgr2=0x7f3a4c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:44.139 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.133+0000 7f3a52375700 1 -- 172.21.15.163:0/2168998806 shutdown_connections 2023-11-15T04:54:44.139 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.133+0000 7f3a52375700 1 -- 172.21.15.163:0/2168998806 wait complete. 2023-11-15T04:54:44.140 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.134+0000 7f3a52375700 1 Processor -- start 2023-11-15T04:54:44.140 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.134+0000 7f3a52375700 1 -- start start 2023-11-15T04:54:44.140 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.134+0000 7f3a52375700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a4c0fab80 0x7f3a4c10ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:44.140 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.134+0000 7f3a52375700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c10d1b0 con 0x7f3a4c0fab80 2023-11-15T04:54:44.140 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.134+0000 7f3a4bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a4c0fab80 0x7f3a4c10ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:44.141 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.134+0000 7f3a4bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a4c0fab80 0x7f3a4c10ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36334/0 (socket says 172.21.15.163:36334) 2023-11-15T04:54:44.141 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.134+0000 7f3a4bfff700 1 -- 172.21.15.163:0/198039119 learned_addr learned my addr 172.21.15.163:0/198039119 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:44.141 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.134+0000 7f3a4bfff700 1 -- 172.21.15.163:0/198039119 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3c00b200 con 0x7f3a4c0fab80 2023-11-15T04:54:44.142 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.135+0000 7f3a4bfff700 1 --2- 172.21.15.163:0/198039119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a4c0fab80 0x7f3a4c10ccc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f3a3c00e040 tx=0x7f3a3c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:44.143 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.135+0000 7f3a48ff9700 1 -- 172.21.15.163:0/198039119 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a3c01b410 con 0x7f3a4c0fab80 2023-11-15T04:54:44.143 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.135+0000 7f3a48ff9700 1 -- 172.21.15.163:0/198039119 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3a3c01b9e0 con 0x7f3a4c0fab80 2023-11-15T04:54:44.143 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.135+0000 7f3a52375700 1 -- 172.21.15.163:0/198039119 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a4c10d3b0 con 0x7f3a4c0fab80 2023-11-15T04:54:44.143 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.135+0000 7f3a48ff9700 1 -- 172.21.15.163:0/198039119 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a3c01a820 con 0x7f3a4c0fab80 2023-11-15T04:54:44.143 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.135+0000 7f3a52375700 1 -- 172.21.15.163:0/198039119 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a4c10d7c0 con 0x7f3a4c0fab80 2023-11-15T04:54:44.144 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.136+0000 7f3a48ff9700 1 -- 172.21.15.163:0/198039119 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f3a3c00f070 con 0x7f3a4c0fab80 2023-11-15T04:54:44.144 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.136+0000 7f3a48ff9700 1 --2- 172.21.15.163:0/198039119 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f3a3802fc50 0x7f3a38032110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:44.144 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.136+0000 7f3a48ff9700 1 -- 172.21.15.163:0/198039119 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f3a3c041220 con 0x7f3a4c0fab80 2023-11-15T04:54:44.144 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.136+0000 7f3a4b7fe700 1 --2- 172.21.15.163:0/198039119 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f3a3802fc50 0x7f3a38032110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:44.144 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.136+0000 7f3a52375700 1 -- 172.21.15.163:0/198039119 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a4c0fc890 con 0x7f3a4c0fab80 2023-11-15T04:54:44.145 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.136+0000 7f3a4b7fe700 1 --2- 172.21.15.163:0/198039119 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f3a3802fc50 0x7f3a38032110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f3a400096a0 tx=0x7f3a40006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:44.145 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.141+0000 7f3a48ff9700 1 -- 172.21.15.163:0/198039119 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a3c016070 con 0x7f3a4c0fab80 2023-11-15T04:54:44.347 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.344+0000 7f3a52375700 1 -- 172.21.15.163:0/198039119 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3a4c10e5f0 con 0x7f3a4c0fab80 2023-11-15T04:54:44.347 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.345+0000 7f3a48ff9700 1 -- 172.21.15.163:0/198039119 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7f3a3c025460 con 0x7f3a4c0fab80 2023-11-15T04:54:44.347 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.347 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: { 2023-11-15T04:54:44.348 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "fsid": "01689f7e-8373-11ee-8dbe-212e2dc638e7", 2023-11-15T04:54:44.348 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "health": { 2023-11-15T04:54:44.348 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T04:54:44.348 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T04:54:44.348 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T04:54:44.349 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2023-11-15T04:54:44.349 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T04:54:44.349 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T04:54:44.356 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 0 2023-11-15T04:54:44.356 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: ], 2023-11-15T04:54:44.356 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T04:54:44.357 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "smithi163" 2023-11-15T04:54:44.357 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: ], 2023-11-15T04:54:44.357 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "quorum_age": 7, 2023-11-15T04:54:44.357 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T04:54:44.357 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:54:44.358 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T04:54:44.358 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T04:54:44.358 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2023-11-15T04:54:44.358 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T04:54:44.358 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:54:44.359 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T04:54:44.359 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T04:54:44.359 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T04:54:44.359 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T04:54:44.359 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T04:54:44.359 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T04:54:44.360 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2023-11-15T04:54:44.360 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T04:54:44.360 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T04:54:44.360 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T04:54:44.360 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T04:54:44.361 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T04:54:44.361 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T04:54:44.361 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T04:54:44.361 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T04:54:44.361 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T04:54:44.362 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2023-11-15T04:54:44.362 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T04:54:44.362 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:54:44.362 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T04:54:44.362 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T04:54:44.363 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2023-11-15T04:54:44.363 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T04:54:44.363 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "available": true, 2023-11-15T04:54:44.363 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T04:54:44.363 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T04:54:44.363 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "iostat", 2023-11-15T04:54:44.364 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "restful" 2023-11-15T04:54:44.364 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: ], 2023-11-15T04:54:44.365 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:54:44.365 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2023-11-15T04:54:44.365 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T04:54:44.365 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T04:54:44.365 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "modified": "2023-11-15T04:54:33.468510+0000", 2023-11-15T04:54:44.366 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "services": {} 2023-11-15T04:54:44.366 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2023-11-15T04:54:44.366 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T04:54:44.366 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: } 2023-11-15T04:54:44.366 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.346+0000 7f3a52375700 1 -- 172.21.15.163:0/198039119 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f3a3802fc50 msgr2=0x7f3a38032110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:44.367 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.346+0000 7f3a52375700 1 --2- 172.21.15.163:0/198039119 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f3a3802fc50 0x7f3a38032110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f3a400096a0 tx=0x7f3a40006b40).stop 2023-11-15T04:54:44.367 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.346+0000 7f3a52375700 1 -- 172.21.15.163:0/198039119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a4c0fab80 msgr2=0x7f3a4c10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:44.367 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.346+0000 7f3a52375700 1 --2- 172.21.15.163:0/198039119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a4c0fab80 0x7f3a4c10ccc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f3a3c00e040 tx=0x7f3a3c006ee0).stop 2023-11-15T04:54:44.367 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.346+0000 7f3a52375700 1 -- 172.21.15.163:0/198039119 shutdown_connections 2023-11-15T04:54:44.368 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.346+0000 7f3a52375700 1 --2- 172.21.15.163:0/198039119 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f3a3802fc50 0x7f3a38032110 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:44.368 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.346+0000 7f3a52375700 1 --2- 172.21.15.163:0/198039119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a4c0fab80 0x7f3a4c10ccc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:44.368 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.346+0000 7f3a52375700 1 -- 172.21.15.163:0/198039119 >> 172.21.15.163:0/198039119 conn(0x7f3a4c0f63c0 msgr2=0x7f3a4c070030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:44.368 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.346+0000 7f3a52375700 1 -- 172.21.15.163:0/198039119 shutdown_connections 2023-11-15T04:54:44.368 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:44.346+0000 7f3a52375700 1 -- 172.21.15.163:0/198039119 wait complete. 2023-11-15T04:54:45.101 INFO:teuthology.orchestra.run.smithi163.stderr:mgr is available 2023-11-15T04:54:45.101 INFO:teuthology.orchestra.run.smithi163.stderr:Enabling cephadm module... 2023-11-15T04:54:45.102 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2023-11-15T04:54:45.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:45 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:54:44.049011+0000 mon.smithi163 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi163.qesyrx(active, since 1.00486s) 2023-11-15T04:54:45.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:45 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:44.346336+0000 mon.smithi163 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.163:0/198039119' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T04:54:45.443 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.437+0000 7f96a0f1e700 1 Processor -- start 2023-11-15T04:54:45.443 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.437+0000 7f96a0f1e700 1 -- start start 2023-11-15T04:54:45.444 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.437+0000 7f96a0f1e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f969c0703f0 0x7f969c0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:45.444 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.437+0000 7f96a0f1e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c0fee10 con 0x7f969c0703f0 2023-11-15T04:54:45.444 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.437+0000 7f969a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f969c0703f0 0x7f969c0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:45.444 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.437+0000 7f969a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f969c0703f0 0x7f969c0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36340/0 (socket says 172.21.15.163:36340) 2023-11-15T04:54:45.445 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.437+0000 7f969a59c700 1 -- 172.21.15.163:0/772033180 learned_addr learned my addr 172.21.15.163:0/772033180 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:45.445 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.438+0000 7f969a59c700 1 -- 172.21.15.163:0/772033180 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c0fef50 con 0x7f969c0703f0 2023-11-15T04:54:45.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.438+0000 7f969a59c700 1 --2- 172.21.15.163:0/772033180 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f969c0703f0 0x7f969c0fe920 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f968c00b5f0 tx=0x7f968c006ee0).ready entity=mon.0 client_cookie=eb2c4fc927a6ae77 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:45.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.438+0000 7f969959a700 1 -- 172.21.15.163:0/772033180 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f968c00d600 con 0x7f969c0703f0 2023-11-15T04:54:45.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.438+0000 7f969959a700 1 -- 172.21.15.163:0/772033180 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f968c00dbd0 con 0x7f969c0703f0 2023-11-15T04:54:45.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.438+0000 7f969959a700 1 -- 172.21.15.163:0/772033180 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f968c011c00 con 0x7f969c0703f0 2023-11-15T04:54:45.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.438+0000 7f96a0f1e700 1 -- 172.21.15.163:0/772033180 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f969c0703f0 msgr2=0x7f969c0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:45.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.438+0000 7f96a0f1e700 1 --2- 172.21.15.163:0/772033180 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f969c0703f0 0x7f969c0fe920 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f968c00b5f0 tx=0x7f968c006ee0).stop 2023-11-15T04:54:45.448 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.439+0000 7f96a0f1e700 1 -- 172.21.15.163:0/772033180 shutdown_connections 2023-11-15T04:54:45.448 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.439+0000 7f96a0f1e700 1 --2- 172.21.15.163:0/772033180 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f969c0703f0 0x7f969c0fe920 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:45.449 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.439+0000 7f96a0f1e700 1 -- 172.21.15.163:0/772033180 >> 172.21.15.163:0/772033180 conn(0x7f969c0f63c0 msgr2=0x7f969c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:45.449 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.439+0000 7f96a0f1e700 1 -- 172.21.15.163:0/772033180 shutdown_connections 2023-11-15T04:54:45.449 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.439+0000 7f96a0f1e700 1 -- 172.21.15.163:0/772033180 wait complete. 2023-11-15T04:54:45.450 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.439+0000 7f96a0f1e700 1 Processor -- start 2023-11-15T04:54:45.450 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.439+0000 7f96a0f1e700 1 -- start start 2023-11-15T04:54:45.451 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.440+0000 7f96a0f1e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f969c0703f0 0x7f969c10a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:45.451 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.440+0000 7f96a0f1e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c10aeb0 con 0x7f969c0703f0 2023-11-15T04:54:45.451 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.440+0000 7f969a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f969c0703f0 0x7f969c10a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:45.452 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.440+0000 7f969a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f969c0703f0 0x7f969c10a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36346/0 (socket says 172.21.15.163:36346) 2023-11-15T04:54:45.452 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.440+0000 7f969a59c700 1 -- 172.21.15.163:0/931143540 learned_addr learned my addr 172.21.15.163:0/931143540 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:45.453 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.440+0000 7f969a59c700 1 -- 172.21.15.163:0/931143540 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968c00b200 con 0x7f969c0703f0 2023-11-15T04:54:45.453 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.440+0000 7f969a59c700 1 --2- 172.21.15.163:0/931143540 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f969c0703f0 0x7f969c10a9c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f968c00b190 tx=0x7f968c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:45.453 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.440+0000 7f968affd700 1 -- 172.21.15.163:0/931143540 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f968c01b410 con 0x7f969c0703f0 2023-11-15T04:54:45.453 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.440+0000 7f968affd700 1 -- 172.21.15.163:0/931143540 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f968c01b9e0 con 0x7f969c0703f0 2023-11-15T04:54:45.454 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.440+0000 7f96a0f1e700 1 -- 172.21.15.163:0/931143540 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f969c10b0b0 con 0x7f969c0703f0 2023-11-15T04:54:45.454 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.440+0000 7f968affd700 1 -- 172.21.15.163:0/931143540 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f968c01a880 con 0x7f969c0703f0 2023-11-15T04:54:45.454 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.440+0000 7f96a0f1e700 1 -- 172.21.15.163:0/931143540 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f969c10b4c0 con 0x7f969c0703f0 2023-11-15T04:54:45.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.441+0000 7f96a0f1e700 1 -- 172.21.15.163:0/931143540 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f969c045ad0 con 0x7f969c0703f0 2023-11-15T04:54:45.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.441+0000 7f968affd700 1 -- 172.21.15.163:0/931143540 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f968c00f070 con 0x7f969c0703f0 2023-11-15T04:54:45.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.441+0000 7f968affd700 1 --2- 172.21.15.163:0/931143540 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f968002fca0 0x7f9680032160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.441+0000 7f968affd700 1 -- 172.21.15.163:0/931143540 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f968c041030 con 0x7f969c0703f0 2023-11-15T04:54:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.441+0000 7f9699d9b700 1 --2- 172.21.15.163:0/931143540 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f968002fca0 0x7f9680032160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.442+0000 7f9699d9b700 1 --2- 172.21.15.163:0/931143540 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f968002fca0 0x7f9680032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f96900096a0 tx=0x7f9690006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:45.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.446+0000 7f968affd700 1 -- 172.21.15.163:0/931143540 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f968c0224f0 con 0x7f969c0703f0 2023-11-15T04:54:45.630 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:45.628+0000 7f96a0f1e700 1 -- 172.21.15.163:0/931143540 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f969c0ffbd0 con 0x7f969c0703f0 2023-11-15T04:54:46.048 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.047+0000 7f968affd700 1 -- 172.21.15.163:0/931143540 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f968c022d80 con 0x7f969c0703f0 2023-11-15T04:54:46.049 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.048+0000 7f968affd700 1 -- 172.21.15.163:0/931143540 <== mon.0 v2:172.21.15.163:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v4) v1 ==== 86+0+0 (secure 0 0 0) 0x7f968c0439b0 con 0x7f969c0703f0 2023-11-15T04:54:46.052 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.050+0000 7f96a0f1e700 1 -- 172.21.15.163:0/931143540 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f968002fca0 msgr2=0x7f9680032160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:46.052 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.050+0000 7f96a0f1e700 1 --2- 172.21.15.163:0/931143540 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f968002fca0 0x7f9680032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f96900096a0 tx=0x7f9690006b40).stop 2023-11-15T04:54:46.052 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.050+0000 7f96a0f1e700 1 -- 172.21.15.163:0/931143540 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f969c0703f0 msgr2=0x7f969c10a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:46.052 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.050+0000 7f96a0f1e700 1 --2- 172.21.15.163:0/931143540 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f969c0703f0 0x7f969c10a9c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f968c00b190 tx=0x7f968c006ee0).stop 2023-11-15T04:54:46.053 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.050+0000 7f96a0f1e700 1 -- 172.21.15.163:0/931143540 shutdown_connections 2023-11-15T04:54:46.053 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.050+0000 7f96a0f1e700 1 --2- 172.21.15.163:0/931143540 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f968002fca0 0x7f9680032160 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:46.053 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.050+0000 7f96a0f1e700 1 --2- 172.21.15.163:0/931143540 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f969c0703f0 0x7f969c10a9c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:46.053 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.050+0000 7f96a0f1e700 1 -- 172.21.15.163:0/931143540 >> 172.21.15.163:0/931143540 conn(0x7f969c0f63c0 msgr2=0x7f969c0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:46.053 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.050+0000 7f96a0f1e700 1 -- 172.21.15.163:0/931143540 shutdown_connections 2023-11-15T04:54:46.054 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.050+0000 7f96a0f1e700 1 -- 172.21.15.163:0/931143540 wait complete. 2023-11-15T04:54:46.372 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:46 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:45.630547+0000 mon.smithi163 (mon.0) 24 : audit [INF] from='client.? 172.21.15.163:0/931143540' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-11-15T04:54:46.403 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-15T04:54:46.776 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.773+0000 7fb87f4e5700 1 Processor -- start 2023-11-15T04:54:46.776 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.774+0000 7fb87f4e5700 1 -- start start 2023-11-15T04:54:46.777 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.774+0000 7fb87f4e5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8780724f0 0x7fb878070a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:46.777 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.774+0000 7fb87f4e5700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb878072910 con 0x7fb8780724f0 2023-11-15T04:54:46.777 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.774+0000 7fb87d281700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8780724f0 0x7fb878070a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:46.777 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.774+0000 7fb87d281700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8780724f0 0x7fb878070a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36382/0 (socket says 172.21.15.163:36382) 2023-11-15T04:54:46.777 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.774+0000 7fb87d281700 1 -- 172.21.15.163:0/2031839905 learned_addr learned my addr 172.21.15.163:0/2031839905 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:46.778 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.774+0000 7fb87d281700 1 -- 172.21.15.163:0/2031839905 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb878070f80 con 0x7fb8780724f0 2023-11-15T04:54:46.778 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.774+0000 7fb87d281700 1 --2- 172.21.15.163:0/2031839905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8780724f0 0x7fb878070a90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fb87400b5f0 tx=0x7fb874006ee0).ready entity=mon.0 client_cookie=165996d94e173ff3 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:46.778 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.774+0000 7fb867fff700 1 -- 172.21.15.163:0/2031839905 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb87400d600 con 0x7fb8780724f0 2023-11-15T04:54:46.778 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.775+0000 7fb867fff700 1 -- 172.21.15.163:0/2031839905 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb87400dbd0 con 0x7fb8780724f0 2023-11-15T04:54:46.778 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.775+0000 7fb867fff700 1 -- 172.21.15.163:0/2031839905 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb874011c00 con 0x7fb8780724f0 2023-11-15T04:54:46.781 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.775+0000 7fb87f4e5700 1 -- 172.21.15.163:0/2031839905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8780724f0 msgr2=0x7fb878070a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:46.781 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.775+0000 7fb87f4e5700 1 --2- 172.21.15.163:0/2031839905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8780724f0 0x7fb878070a90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fb87400b5f0 tx=0x7fb874006ee0).stop 2023-11-15T04:54:46.782 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.775+0000 7fb87f4e5700 1 -- 172.21.15.163:0/2031839905 shutdown_connections 2023-11-15T04:54:46.782 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.775+0000 7fb87f4e5700 1 --2- 172.21.15.163:0/2031839905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8780724f0 0x7fb878070a90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:46.782 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.775+0000 7fb87f4e5700 1 -- 172.21.15.163:0/2031839905 >> 172.21.15.163:0/2031839905 conn(0x7fb8780f74c0 msgr2=0x7fb8780f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:46.782 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.776+0000 7fb87f4e5700 1 -- 172.21.15.163:0/2031839905 shutdown_connections 2023-11-15T04:54:46.783 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.776+0000 7fb87f4e5700 1 -- 172.21.15.163:0/2031839905 wait complete. 2023-11-15T04:54:46.783 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.776+0000 7fb87f4e5700 1 Processor -- start 2023-11-15T04:54:46.783 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.776+0000 7fb87f4e5700 1 -- start start 2023-11-15T04:54:46.783 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.776+0000 7fb87f4e5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8780724f0 0x7fb87810cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:46.783 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.776+0000 7fb87f4e5700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87810d1d0 con 0x7fb8780724f0 2023-11-15T04:54:46.783 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.776+0000 7fb87d281700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8780724f0 0x7fb87810cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:46.784 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.776+0000 7fb87d281700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8780724f0 0x7fb87810cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36398/0 (socket says 172.21.15.163:36398) 2023-11-15T04:54:46.784 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.776+0000 7fb87d281700 1 -- 172.21.15.163:0/549217266 learned_addr learned my addr 172.21.15.163:0/549217266 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:46.784 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.777+0000 7fb87d281700 1 -- 172.21.15.163:0/549217266 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87400b200 con 0x7fb8780724f0 2023-11-15T04:54:46.784 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.777+0000 7fb87d281700 1 --2- 172.21.15.163:0/549217266 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8780724f0 0x7fb87810cce0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fb874011820 tx=0x7fb874006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:46.784 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.777+0000 7fb865ffb700 1 -- 172.21.15.163:0/549217266 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb87401b410 con 0x7fb8780724f0 2023-11-15T04:54:46.785 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.777+0000 7fb87f4e5700 1 -- 172.21.15.163:0/549217266 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87810d3d0 con 0x7fb8780724f0 2023-11-15T04:54:46.785 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.777+0000 7fb865ffb700 1 -- 172.21.15.163:0/549217266 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb87401b9e0 con 0x7fb8780724f0 2023-11-15T04:54:46.785 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.777+0000 7fb865ffb700 1 -- 172.21.15.163:0/549217266 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb87401a8b0 con 0x7fb8780724f0 2023-11-15T04:54:46.785 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.777+0000 7fb87f4e5700 1 -- 172.21.15.163:0/549217266 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb87810d7e0 con 0x7fb8780724f0 2023-11-15T04:54:46.785 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.778+0000 7fb87f4e5700 1 -- 172.21.15.163:0/549217266 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb878045ad0 con 0x7fb8780724f0 2023-11-15T04:54:46.786 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.778+0000 7fb865ffb700 1 -- 172.21.15.163:0/549217266 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb87400f070 con 0x7fb8780724f0 2023-11-15T04:54:46.786 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.779+0000 7fb865ffb700 1 --2- 172.21.15.163:0/549217266 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7fb86002fdb0 0x7fb860032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:46.786 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.779+0000 7fb865ffb700 1 -- 172.21.15.163:0/549217266 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb874041220 con 0x7fb8780724f0 2023-11-15T04:54:46.786 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.779+0000 7fb87ca80700 1 -- 172.21.15.163:0/549217266 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7fb86002fdb0 msgr2=0x7fb860032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.163:6800/1837204690 2023-11-15T04:54:46.786 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.779+0000 7fb87ca80700 1 --2- 172.21.15.163:0/549217266 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7fb86002fdb0 0x7fb860032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T04:54:46.787 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.783+0000 7fb865ffb700 1 -- 172.21.15.163:0/549217266 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb874022d20 con 0x7fb8780724f0 2023-11-15T04:54:46.982 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.980+0000 7fb87ca80700 1 -- 172.21.15.163:0/549217266 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7fb86002fdb0 msgr2=0x7fb860032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.163:6800/1837204690 2023-11-15T04:54:46.982 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.980+0000 7fb87ca80700 1 --2- 172.21.15.163:0/549217266 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7fb86002fdb0 0x7fb860032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T04:54:46.983 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.980+0000 7fb87f4e5700 1 -- 172.21.15.163:0/549217266 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fb878106620 con 0x7fb8780724f0 2023-11-15T04:54:46.983 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.980+0000 7fb865ffb700 1 -- 172.21.15.163:0/549217266 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v4) v1 ==== 56+0+103 (secure 0 0 0) 0x7fb874016070 con 0x7fb8780724f0 2023-11-15T04:54:46.983 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: { 2023-11-15T04:54:46.983 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 4, 2023-11-15T04:54:46.983 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "available": true, 2023-11-15T04:54:46.984 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "active_name": "smithi163.qesyrx", 2023-11-15T04:54:46.984 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-15T04:54:46.984 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: } 2023-11-15T04:54:46.985 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.982+0000 7fb87f4e5700 1 -- 172.21.15.163:0/549217266 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7fb86002fdb0 msgr2=0x7fb860032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T04:54:46.986 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.982+0000 7fb87f4e5700 1 --2- 172.21.15.163:0/549217266 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7fb86002fdb0 0x7fb860032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:54:46.986 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.982+0000 7fb87f4e5700 1 -- 172.21.15.163:0/549217266 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8780724f0 msgr2=0x7fb87810cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:46.986 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.982+0000 7fb87f4e5700 1 --2- 172.21.15.163:0/549217266 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8780724f0 0x7fb87810cce0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fb874011820 tx=0x7fb874006ee0).stop 2023-11-15T04:54:46.986 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.982+0000 7fb87f4e5700 1 -- 172.21.15.163:0/549217266 shutdown_connections 2023-11-15T04:54:46.986 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.982+0000 7fb87f4e5700 1 --2- 172.21.15.163:0/549217266 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7fb86002fdb0 0x7fb860032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:54:46.987 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.982+0000 7fb87f4e5700 1 --2- 172.21.15.163:0/549217266 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8780724f0 0x7fb87810cce0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:46.987 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.982+0000 7fb87f4e5700 1 -- 172.21.15.163:0/549217266 >> 172.21.15.163:0/549217266 conn(0x7fb8780f74c0 msgr2=0x7fb8780f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:46.987 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.982+0000 7fb87f4e5700 1 -- 172.21.15.163:0/549217266 shutdown_connections 2023-11-15T04:54:46.987 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:46.982+0000 7fb87f4e5700 1 -- 172.21.15.163:0/549217266 wait complete. 2023-11-15T04:54:47.290 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:47 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:46.049610+0000 mon.smithi163 (mon.0) 25 : audit [INF] from='client.? 172.21.15.163:0/931143540' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-11-15T04:54:47.290 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:47 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:54:46.049683+0000 mon.smithi163 (mon.0) 26 : cluster [DBG] mgrmap e4: smithi163.qesyrx(active, since 3s) 2023-11-15T04:54:47.291 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:47 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:46.981953+0000 mon.smithi163 (mon.0) 27 : audit [DBG] from='client.? 172.21.15.163:0/549217266' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-15T04:54:47.331 INFO:teuthology.orchestra.run.smithi163.stderr:Waiting for the mgr to restart... 2023-11-15T04:54:47.331 INFO:teuthology.orchestra.run.smithi163.stderr:Waiting for mgr epoch 4... 2023-11-15T04:54:47.331 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-15T04:54:47.727 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.725+0000 7f90fe83a700 1 Processor -- start 2023-11-15T04:54:47.728 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.725+0000 7f90fe83a700 1 -- start start 2023-11-15T04:54:47.728 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.725+0000 7f90fe83a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90f8070430 0x7f90f80fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:47.728 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.725+0000 7f90fe83a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f80fedd0 con 0x7f90f8070430 2023-11-15T04:54:47.728 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.725+0000 7f90f7fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90f8070430 0x7f90f80fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:47.729 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.725+0000 7f90f7fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90f8070430 0x7f90f80fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36400/0 (socket says 172.21.15.163:36400) 2023-11-15T04:54:47.729 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.725+0000 7f90f7fff700 1 -- 172.21.15.163:0/4254737853 learned_addr learned my addr 172.21.15.163:0/4254737853 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:47.729 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.726+0000 7f90f7fff700 1 -- 172.21.15.163:0/4254737853 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f80fef10 con 0x7f90f8070430 2023-11-15T04:54:47.729 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.726+0000 7f90f7fff700 1 --2- 172.21.15.163:0/4254737853 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90f8070430 0x7f90f80fe8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f90e800b5f0 tx=0x7f90e8006ee0).ready entity=mon.0 client_cookie=418622fb94ae1902 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:47.729 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.726+0000 7f90f6ffd700 1 -- 172.21.15.163:0/4254737853 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90e800d600 con 0x7f90f8070430 2023-11-15T04:54:47.730 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.726+0000 7f90f6ffd700 1 -- 172.21.15.163:0/4254737853 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f90e800dbd0 con 0x7f90f8070430 2023-11-15T04:54:47.730 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.726+0000 7f90f6ffd700 1 -- 172.21.15.163:0/4254737853 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90e8011c00 con 0x7f90f8070430 2023-11-15T04:54:47.733 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.727+0000 7f90fe83a700 1 -- 172.21.15.163:0/4254737853 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90f8070430 msgr2=0x7f90f80fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:47.733 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.727+0000 7f90fe83a700 1 --2- 172.21.15.163:0/4254737853 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90f8070430 0x7f90f80fe8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f90e800b5f0 tx=0x7f90e8006ee0).stop 2023-11-15T04:54:47.733 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.727+0000 7f90fe83a700 1 -- 172.21.15.163:0/4254737853 shutdown_connections 2023-11-15T04:54:47.733 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.727+0000 7f90fe83a700 1 --2- 172.21.15.163:0/4254737853 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90f8070430 0x7f90f80fe8e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:47.734 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.727+0000 7f90fe83a700 1 -- 172.21.15.163:0/4254737853 >> 172.21.15.163:0/4254737853 conn(0x7f90f80f63c0 msgr2=0x7f90f80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:47.734 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.727+0000 7f90fe83a700 1 -- 172.21.15.163:0/4254737853 shutdown_connections 2023-11-15T04:54:47.734 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.727+0000 7f90fe83a700 1 -- 172.21.15.163:0/4254737853 wait complete. 2023-11-15T04:54:47.734 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.727+0000 7f90fe83a700 1 Processor -- start 2023-11-15T04:54:47.734 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.728+0000 7f90fe83a700 1 -- start start 2023-11-15T04:54:47.735 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.728+0000 7f90fe83a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90f8070430 0x7f90f80fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:47.735 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.728+0000 7f90fe83a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f80fa7e0 con 0x7f90f8070430 2023-11-15T04:54:47.735 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.728+0000 7f90f7fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90f8070430 0x7f90f80fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:47.735 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.728+0000 7f90f7fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90f8070430 0x7f90f80fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36404/0 (socket says 172.21.15.163:36404) 2023-11-15T04:54:47.735 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.728+0000 7f90f7fff700 1 -- 172.21.15.163:0/3510959309 learned_addr learned my addr 172.21.15.163:0/3510959309 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:47.736 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.728+0000 7f90f7fff700 1 -- 172.21.15.163:0/3510959309 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e800b200 con 0x7f90f8070430 2023-11-15T04:54:47.736 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.728+0000 7f90f7fff700 1 --2- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90f8070430 0x7f90f80fc360 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f90e800b190 tx=0x7f90e8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:47.736 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.729+0000 7f90f4ff9700 1 -- 172.21.15.163:0/3510959309 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90e801b410 con 0x7f90f8070430 2023-11-15T04:54:47.736 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.729+0000 7f90f4ff9700 1 -- 172.21.15.163:0/3510959309 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f90e801b9e0 con 0x7f90f8070430 2023-11-15T04:54:47.736 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.729+0000 7f90fe83a700 1 -- 172.21.15.163:0/3510959309 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f80fa980 con 0x7f90f8070430 2023-11-15T04:54:47.737 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.729+0000 7f90f4ff9700 1 -- 172.21.15.163:0/3510959309 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90e801a880 con 0x7f90f8070430 2023-11-15T04:54:47.737 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.729+0000 7f90fe83a700 1 -- 172.21.15.163:0/3510959309 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f80fad90 con 0x7f90f8070430 2023-11-15T04:54:47.737 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.730+0000 7f90f4ff9700 1 -- 172.21.15.163:0/3510959309 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f90e800f070 con 0x7f90f8070430 2023-11-15T04:54:47.737 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.730+0000 7f90f4ff9700 1 --2- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f90e402fdb0 0x7f90e4032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:47.737 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.730+0000 7f90f4ff9700 1 -- 172.21.15.163:0/3510959309 --> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90e4032930 con 0x7f90e402fdb0 2023-11-15T04:54:47.738 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.730+0000 7f90f4ff9700 1 -- 172.21.15.163:0/3510959309 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f90e8041fc0 con 0x7f90f8070430 2023-11-15T04:54:47.738 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.730+0000 7f90f77fe700 1 -- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f90e402fdb0 msgr2=0x7f90e4032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.163:6800/1837204690 2023-11-15T04:54:47.738 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.730+0000 7f90f77fe700 1 --2- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f90e402fdb0 0x7f90e4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T04:54:47.932 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.931+0000 7f90f77fe700 1 -- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f90e402fdb0 msgr2=0x7f90e4032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.163:6800/1837204690 2023-11-15T04:54:47.933 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:47.931+0000 7f90f77fe700 1 --2- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f90e402fdb0 0x7f90e4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T04:54:48.333 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:48.332+0000 7f90f77fe700 1 -- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f90e402fdb0 msgr2=0x7f90e4032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.163:6800/1837204690 2023-11-15T04:54:48.334 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:48.332+0000 7f90f77fe700 1 --2- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f90e402fdb0 0x7f90e4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T04:54:49.135 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:49.133+0000 7f90f77fe700 1 -- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f90e402fdb0 msgr2=0x7f90e4032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.163:6800/1837204690 2023-11-15T04:54:49.135 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:49.133+0000 7f90f77fe700 1 --2- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f90e402fdb0 0x7f90e4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T04:54:50.735 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:50.735+0000 7f90f77fe700 1 -- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f90e402fdb0 msgr2=0x7f90e4032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.163:6800/1837204690 2023-11-15T04:54:50.736 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:50.735+0000 7f90f77fe700 1 --2- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f90e402fdb0 0x7f90e4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-15T04:54:51.042 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:51.042+0000 7f90f4ff9700 1 -- 172.21.15.163:0/3510959309 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mgrmap(e 5) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f90e801a9e0 con 0x7f90f8070430 2023-11-15T04:54:51.043 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:51.042+0000 7f90f4ff9700 1 -- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f90e402fdb0 msgr2=0x7f90e4032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T04:54:51.043 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:51.042+0000 7f90f4ff9700 1 --2- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f90e402fdb0 0x7f90e4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:54:51.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:51 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:54:50.990405+0000 mon.smithi163 (mon.0) 28 : cluster [INF] Active manager daemon smithi163.qesyrx restarted 2023-11-15T04:54:51.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:51 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:54:50.990786+0000 mon.smithi163 (mon.0) 29 : cluster [INF] Activating manager daemon smithi163.qesyrx 2023-11-15T04:54:51.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:51 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:54:50.992563+0000 mon.smithi163 (mon.0) 30 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-11-15T04:54:52.050 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.049+0000 7f90f4ff9700 1 -- 172.21.15.163:0/3510959309 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f90e80257e0 con 0x7f90f8070430 2023-11-15T04:54:52.051 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.049+0000 7f90f4ff9700 1 --2- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f90e4033880 0x7f90e4035c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:52.052 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.050+0000 7f90f4ff9700 1 -- 172.21.15.163:0/3510959309 --> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90e4032930 con 0x7f90e4033880 2023-11-15T04:54:52.052 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.050+0000 7f90f77fe700 1 --2- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f90e4033880 0x7f90e4035c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:52.052 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.051+0000 7f90f77fe700 1 --2- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f90e4033880 0x7f90e4035c70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f90ec006fd0 tx=0x7f90ec006d70).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:52.052 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.051+0000 7f90f4ff9700 1 -- 172.21.15.163:0/3510959309 <== mgr.14114 v2:172.21.15.163:6800/1606811720 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f90e4032930 con 0x7f90e4033880 2023-11-15T04:54:52.058 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.058+0000 7f90fe83a700 1 -- 172.21.15.163:0/3510959309 --> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f90f8059100 con 0x7f90e4033880 2023-11-15T04:54:52.060 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.059+0000 7f90f4ff9700 1 -- 172.21.15.163:0/3510959309 <== mgr.14114 v2:172.21.15.163:6800/1606811720 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f90f8059100 con 0x7f90e4033880 2023-11-15T04:54:52.060 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: { 2023-11-15T04:54:52.060 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "mgrmap_epoch": 6, 2023-11-15T04:54:52.061 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "initialized": true 2023-11-15T04:54:52.061 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: } 2023-11-15T04:54:52.061 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.059+0000 7f90fe83a700 1 -- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f90e4033880 msgr2=0x7f90e4035c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:52.061 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.059+0000 7f90fe83a700 1 --2- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f90e4033880 0x7f90e4035c70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f90ec006fd0 tx=0x7f90ec006d70).stop 2023-11-15T04:54:52.061 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.059+0000 7f90fe83a700 1 -- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90f8070430 msgr2=0x7f90f80fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:52.062 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.059+0000 7f90fe83a700 1 --2- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90f8070430 0x7f90f80fc360 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f90e800b190 tx=0x7f90e8006ee0).stop 2023-11-15T04:54:52.062 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.059+0000 7f90fe83a700 1 -- 172.21.15.163:0/3510959309 shutdown_connections 2023-11-15T04:54:52.062 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.059+0000 7f90fe83a700 1 --2- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f90e4033880 0x7f90e4035c70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:52.063 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.059+0000 7f90fe83a700 1 --2- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90f8070430 0x7f90f80fc360 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:52.063 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.059+0000 7f90fe83a700 1 --2- 172.21.15.163:0/3510959309 >> [v2:172.21.15.163:6800/1837204690,v1:172.21.15.163:6801/1837204690] conn(0x7f90e402fdb0 0x7f90e4032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:54:52.063 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.059+0000 7f90fe83a700 1 -- 172.21.15.163:0/3510959309 >> 172.21.15.163:0/3510959309 conn(0x7f90f80f63c0 msgr2=0x7f90f80f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:52.064 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.059+0000 7f90fe83a700 1 -- 172.21.15.163:0/3510959309 shutdown_connections 2023-11-15T04:54:52.064 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.059+0000 7f90fe83a700 1 -- 172.21.15.163:0/3510959309 wait complete. 2023-11-15T04:54:52.363 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:52 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:54:51.043691+0000 mon.smithi163 (mon.0) 31 : cluster [DBG] mgrmap e5: smithi163.qesyrx(active, starting, since 0.0529791s) 2023-11-15T04:54:52.364 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:52 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:51.045427+0000 mon.smithi163 (mon.0) 32 : audit [DBG] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2023-11-15T04:54:52.364 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:52 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:51.046012+0000 mon.smithi163 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "mgr metadata", "who": "smithi163.qesyrx", "id": "smithi163.qesyrx"}]: dispatch 2023-11-15T04:54:52.364 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:52 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:51.046700+0000 mon.smithi163 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T04:54:52.364 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:52 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:51.046836+0000 mon.smithi163 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T04:54:52.364 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:52 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:51.046972+0000 mon.smithi163 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T04:54:52.365 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:52 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:54:51.061095+0000 mon.smithi163 (mon.0) 37 : cluster [INF] Manager daemon smithi163.qesyrx is now available 2023-11-15T04:54:52.365 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:52 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:51.076164+0000 mon.smithi163 (mon.0) 38 : audit [INF] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' 2023-11-15T04:54:52.365 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:52 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:51.081454+0000 mon.smithi163 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' 2023-11-15T04:54:52.365 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:52 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:51.087116+0000 mon.smithi163 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' 2023-11-15T04:54:52.365 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:52 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:51.176281+0000 mon.smithi163 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' 2023-11-15T04:54:52.366 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:52 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:51.176950+0000 mon.smithi163 (mon.0) 42 : audit [DBG] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:54:52.366 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:52 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:51.178004+0000 mon.smithi163 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:54:52.366 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:52 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:51.179422+0000 mon.smithi163 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:54:52.366 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:52 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:51.189580+0000 mon.smithi163 (mon.0) 45 : audit [INF] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:54:52.366 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:52 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:51.190714+0000 mon.smithi163 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/trash_purge_schedule"}]: dispatch 2023-11-15T04:54:52.394 INFO:teuthology.orchestra.run.smithi163.stderr:mgr epoch 4 is available 2023-11-15T04:54:52.394 INFO:teuthology.orchestra.run.smithi163.stderr:Setting orchestrator backend to cephadm... 2023-11-15T04:54:52.395 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2023-11-15T04:54:52.726 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.721+0000 7f421edc8700 1 Processor -- start 2023-11-15T04:54:52.727 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.721+0000 7f421edc8700 1 -- start start 2023-11-15T04:54:52.727 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.721+0000 7f421edc8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42180fbcb0 0x7f42180fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:52.727 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.721+0000 7f421edc8700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42180fc5c0 con 0x7f42180fbcb0 2023-11-15T04:54:52.727 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.721+0000 7f421cb64700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42180fbcb0 0x7f42180fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:52.728 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.721+0000 7f421cb64700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42180fbcb0 0x7f42180fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34870/0 (socket says 172.21.15.163:34870) 2023-11-15T04:54:52.728 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.721+0000 7f421cb64700 1 -- 172.21.15.163:0/30033100 learned_addr learned my addr 172.21.15.163:0/30033100 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:52.728 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.721+0000 7f421cb64700 1 -- 172.21.15.163:0/30033100 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42180fc700 con 0x7f42180fbcb0 2023-11-15T04:54:52.728 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.722+0000 7f421cb64700 1 --2- 172.21.15.163:0/30033100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42180fbcb0 0x7f42180fc0d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f420c00b5f0 tx=0x7f420c006ee0).ready entity=mon.0 client_cookie=68d5d600ba53d16c server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:52.728 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.722+0000 7f42177fe700 1 -- 172.21.15.163:0/30033100 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f420c00d600 con 0x7f42180fbcb0 2023-11-15T04:54:52.729 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.722+0000 7f42177fe700 1 -- 172.21.15.163:0/30033100 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f420c00dbd0 con 0x7f42180fbcb0 2023-11-15T04:54:52.729 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.722+0000 7f42177fe700 1 -- 172.21.15.163:0/30033100 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f420c011c00 con 0x7f42180fbcb0 2023-11-15T04:54:52.729 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.722+0000 7f421edc8700 1 -- 172.21.15.163:0/30033100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42180fbcb0 msgr2=0x7f42180fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:52.729 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.722+0000 7f421edc8700 1 --2- 172.21.15.163:0/30033100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42180fbcb0 0x7f42180fc0d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f420c00b5f0 tx=0x7f420c006ee0).stop 2023-11-15T04:54:52.729 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.722+0000 7f421edc8700 1 -- 172.21.15.163:0/30033100 shutdown_connections 2023-11-15T04:54:52.730 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.722+0000 7f421edc8700 1 --2- 172.21.15.163:0/30033100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42180fbcb0 0x7f42180fc0d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:52.730 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.722+0000 7f421edc8700 1 -- 172.21.15.163:0/30033100 >> 172.21.15.163:0/30033100 conn(0x7f42180f74b0 msgr2=0x7f42180f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:52.730 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.722+0000 7f421edc8700 1 -- 172.21.15.163:0/30033100 shutdown_connections 2023-11-15T04:54:52.730 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.722+0000 7f421edc8700 1 -- 172.21.15.163:0/30033100 wait complete. 2023-11-15T04:54:52.730 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.723+0000 7f421edc8700 1 Processor -- start 2023-11-15T04:54:52.731 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.723+0000 7f421edc8700 1 -- start start 2023-11-15T04:54:52.731 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.723+0000 7f421edc8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42180fbcb0 0x7f421810aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:52.731 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.724+0000 7f421edc8700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421810af00 con 0x7f42180fbcb0 2023-11-15T04:54:52.731 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.724+0000 7f421cb64700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42180fbcb0 0x7f421810aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:52.731 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.724+0000 7f421cb64700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42180fbcb0 0x7f421810aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34884/0 (socket says 172.21.15.163:34884) 2023-11-15T04:54:52.732 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.724+0000 7f421cb64700 1 -- 172.21.15.163:0/958658889 learned_addr learned my addr 172.21.15.163:0/958658889 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:52.732 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.724+0000 7f421cb64700 1 -- 172.21.15.163:0/958658889 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420c00b200 con 0x7f42180fbcb0 2023-11-15T04:54:52.732 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.724+0000 7f421cb64700 1 --2- 172.21.15.163:0/958658889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42180fbcb0 0x7f421810aa10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f420c00b1d0 tx=0x7f420c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:52.732 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.724+0000 7f42157fa700 1 -- 172.21.15.163:0/958658889 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f420c01b410 con 0x7f42180fbcb0 2023-11-15T04:54:52.733 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.724+0000 7f42157fa700 1 -- 172.21.15.163:0/958658889 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f420c01b9e0 con 0x7f42180fbcb0 2023-11-15T04:54:52.733 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.724+0000 7f42157fa700 1 -- 172.21.15.163:0/958658889 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f420c01a840 con 0x7f42180fbcb0 2023-11-15T04:54:52.734 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.724+0000 7f421edc8700 1 -- 172.21.15.163:0/958658889 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f421810b100 con 0x7f42180fbcb0 2023-11-15T04:54:52.734 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.724+0000 7f421edc8700 1 -- 172.21.15.163:0/958658889 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f421810b510 con 0x7f42180fbcb0 2023-11-15T04:54:52.734 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.725+0000 7f42157fa700 1 -- 172.21.15.163:0/958658889 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f420c00f070 con 0x7f42180fbcb0 2023-11-15T04:54:52.734 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.725+0000 7f42157fa700 1 --2- 172.21.15.163:0/958658889 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f41fc02fcf0 0x7f41fc0321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:52.734 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.725+0000 7f42157fa700 1 -- 172.21.15.163:0/958658889 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f420c0411b0 con 0x7f42180fbcb0 2023-11-15T04:54:52.735 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.725+0000 7f4217fff700 1 --2- 172.21.15.163:0/958658889 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f41fc02fcf0 0x7f41fc0321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:52.735 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.725+0000 7f421edc8700 1 -- 172.21.15.163:0/958658889 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4218045ad0 con 0x7f42180fbcb0 2023-11-15T04:54:52.735 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.726+0000 7f4217fff700 1 --2- 172.21.15.163:0/958658889 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f41fc02fcf0 0x7f41fc0321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f42040096a0 tx=0x7f4204006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:52.736 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.730+0000 7f42157fa700 1 -- 172.21.15.163:0/958658889 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f420c025460 con 0x7f42180fbcb0 2023-11-15T04:54:52.907 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.906+0000 7f421edc8700 1 -- 172.21.15.163:0/958658889 --> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f4218104960 con 0x7f41fc02fcf0 2023-11-15T04:54:52.911 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.911+0000 7f42157fa700 1 -- 172.21.15.163:0/958658889 <== mgr.14114 v2:172.21.15.163:6800/1606811720 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4218104960 con 0x7f41fc02fcf0 2023-11-15T04:54:52.914 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.913+0000 7f421edc8700 1 -- 172.21.15.163:0/958658889 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f41fc02fcf0 msgr2=0x7f41fc0321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:52.915 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.913+0000 7f421edc8700 1 --2- 172.21.15.163:0/958658889 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f41fc02fcf0 0x7f41fc0321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f42040096a0 tx=0x7f4204006b40).stop 2023-11-15T04:54:52.915 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.913+0000 7f421edc8700 1 -- 172.21.15.163:0/958658889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42180fbcb0 msgr2=0x7f421810aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:52.915 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.913+0000 7f421edc8700 1 --2- 172.21.15.163:0/958658889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42180fbcb0 0x7f421810aa10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f420c00b1d0 tx=0x7f420c006ee0).stop 2023-11-15T04:54:52.915 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.913+0000 7f421edc8700 1 -- 172.21.15.163:0/958658889 shutdown_connections 2023-11-15T04:54:52.915 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.913+0000 7f421edc8700 1 --2- 172.21.15.163:0/958658889 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f41fc02fcf0 0x7f41fc0321b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:52.916 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.913+0000 7f421edc8700 1 --2- 172.21.15.163:0/958658889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42180fbcb0 0x7f421810aa10 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:52.916 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.913+0000 7f421edc8700 1 -- 172.21.15.163:0/958658889 >> 172.21.15.163:0/958658889 conn(0x7f42180f74b0 msgr2=0x7f42180f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:52.916 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.914+0000 7f421edc8700 1 -- 172.21.15.163:0/958658889 shutdown_connections 2023-11-15T04:54:52.916 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:52.914+0000 7f421edc8700 1 -- 172.21.15.163:0/958658889 wait complete. 2023-11-15T04:54:53.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:53 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:52.051807+0000 mgr.smithi163.qesyrx (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-15T04:54:53.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:53 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:54:52.052549+0000 mon.smithi163 (mon.0) 47 : cluster [DBG] mgrmap e6: smithi163.qesyrx(active, since 1.06183s) 2023-11-15T04:54:53.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:53 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:52.059352+0000 mgr.smithi163.qesyrx (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-15T04:54:53.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:53 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:52.911645+0000 mon.smithi163 (mon.0) 48 : audit [INF] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' 2023-11-15T04:54:53.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:53 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:52.915200+0000 mon.smithi163 (mon.0) 49 : audit [DBG] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:54:53.285 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2023-11-15T04:54:53.651 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.647+0000 7fb551239700 1 Processor -- start 2023-11-15T04:54:53.651 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.647+0000 7fb551239700 1 -- start start 2023-11-15T04:54:53.651 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.647+0000 7fb551239700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb54c0703f0 0x7fb54c0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:53.654 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.647+0000 7fb551239700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c0fee10 con 0x7fb54c0703f0 2023-11-15T04:54:53.654 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.647+0000 7fb54ad9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb54c0703f0 0x7fb54c0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:53.655 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.647+0000 7fb54ad9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb54c0703f0 0x7fb54c0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34892/0 (socket says 172.21.15.163:34892) 2023-11-15T04:54:53.655 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.647+0000 7fb54ad9d700 1 -- 172.21.15.163:0/323831323 learned_addr learned my addr 172.21.15.163:0/323831323 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:53.656 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.647+0000 7fb54ad9d700 1 -- 172.21.15.163:0/323831323 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54c0fef50 con 0x7fb54c0703f0 2023-11-15T04:54:53.656 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.647+0000 7fb54ad9d700 1 --2- 172.21.15.163:0/323831323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb54c0703f0 0x7fb54c0fe920 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fb53c00b5f0 tx=0x7fb53c006ee0).ready entity=mon.0 client_cookie=5c576669eaad3df5 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:53.656 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.648+0000 7fb549d9b700 1 -- 172.21.15.163:0/323831323 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb53c00d600 con 0x7fb54c0703f0 2023-11-15T04:54:53.657 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.648+0000 7fb549d9b700 1 -- 172.21.15.163:0/323831323 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb53c00dbd0 con 0x7fb54c0703f0 2023-11-15T04:54:53.657 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.648+0000 7fb549d9b700 1 -- 172.21.15.163:0/323831323 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb53c011c00 con 0x7fb54c0703f0 2023-11-15T04:54:53.657 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.648+0000 7fb551239700 1 -- 172.21.15.163:0/323831323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb54c0703f0 msgr2=0x7fb54c0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:53.658 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.648+0000 7fb551239700 1 --2- 172.21.15.163:0/323831323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb54c0703f0 0x7fb54c0fe920 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fb53c00b5f0 tx=0x7fb53c006ee0).stop 2023-11-15T04:54:53.658 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.648+0000 7fb551239700 1 -- 172.21.15.163:0/323831323 shutdown_connections 2023-11-15T04:54:53.659 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.648+0000 7fb551239700 1 --2- 172.21.15.163:0/323831323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb54c0703f0 0x7fb54c0fe920 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:53.659 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.648+0000 7fb551239700 1 -- 172.21.15.163:0/323831323 >> 172.21.15.163:0/323831323 conn(0x7fb54c0f63c0 msgr2=0x7fb54c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:53.659 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.648+0000 7fb551239700 1 -- 172.21.15.163:0/323831323 shutdown_connections 2023-11-15T04:54:53.660 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.648+0000 7fb551239700 1 -- 172.21.15.163:0/323831323 wait complete. 2023-11-15T04:54:53.660 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.649+0000 7fb551239700 1 Processor -- start 2023-11-15T04:54:53.660 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.649+0000 7fb551239700 1 -- start start 2023-11-15T04:54:53.661 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.649+0000 7fb551239700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb54c0703f0 0x7fb54c0fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:53.661 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.649+0000 7fb551239700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c0fa820 con 0x7fb54c0703f0 2023-11-15T04:54:53.662 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.649+0000 7fb54ad9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb54c0703f0 0x7fb54c0fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:53.662 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.649+0000 7fb54ad9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb54c0703f0 0x7fb54c0fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34908/0 (socket says 172.21.15.163:34908) 2023-11-15T04:54:53.663 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.649+0000 7fb54ad9d700 1 -- 172.21.15.163:0/1376640604 learned_addr learned my addr 172.21.15.163:0/1376640604 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:53.663 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.649+0000 7fb54ad9d700 1 -- 172.21.15.163:0/1376640604 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53c00b200 con 0x7fb54c0703f0 2023-11-15T04:54:53.663 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.650+0000 7fb54ad9d700 1 --2- 172.21.15.163:0/1376640604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb54c0703f0 0x7fb54c0fc3a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fb53c00e040 tx=0x7fb53c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:53.664 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.650+0000 7fb5337fe700 1 -- 172.21.15.163:0/1376640604 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb53c00d9f0 con 0x7fb54c0703f0 2023-11-15T04:54:53.664 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.650+0000 7fb551239700 1 -- 172.21.15.163:0/1376640604 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb54c0fa9c0 con 0x7fb54c0703f0 2023-11-15T04:54:53.664 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.650+0000 7fb5337fe700 1 -- 172.21.15.163:0/1376640604 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb53c01a9f0 con 0x7fb54c0703f0 2023-11-15T04:54:53.665 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.650+0000 7fb5337fe700 1 -- 172.21.15.163:0/1376640604 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb53c01b8c0 con 0x7fb54c0703f0 2023-11-15T04:54:53.665 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.650+0000 7fb551239700 1 -- 172.21.15.163:0/1376640604 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb54c0fadd0 con 0x7fb54c0703f0 2023-11-15T04:54:53.666 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.651+0000 7fb5337fe700 1 -- 172.21.15.163:0/1376640604 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7fb53c00f070 con 0x7fb54c0703f0 2023-11-15T04:54:53.667 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.651+0000 7fb551239700 1 -- 172.21.15.163:0/1376640604 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb54c045ad0 con 0x7fb54c0703f0 2023-11-15T04:54:53.668 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.651+0000 7fb5337fe700 1 --2- 172.21.15.163:0/1376640604 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fb53802fd10 0x7fb5380321d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:53.668 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.651+0000 7fb5337fe700 1 -- 172.21.15.163:0/1376640604 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb53c0411c0 con 0x7fb54c0703f0 2023-11-15T04:54:53.668 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.651+0000 7fb54a59c700 1 --2- 172.21.15.163:0/1376640604 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fb53802fd10 0x7fb5380321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:53.669 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.651+0000 7fb54a59c700 1 --2- 172.21.15.163:0/1376640604 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fb53802fd10 0x7fb5380321d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb5400096a0 tx=0x7fb540006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:53.669 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.655+0000 7fb5337fe700 1 -- 172.21.15.163:0/1376640604 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb53c022d20 con 0x7fb54c0703f0 2023-11-15T04:54:53.807 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.807+0000 7fb551239700 1 -- 172.21.15.163:0/1376640604 --> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fb54c0fb010 con 0x7fb53802fd10 2023-11-15T04:54:53.808 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.807+0000 7fb5337fe700 1 -- 172.21.15.163:0/1376640604 <== mgr.14114 v2:172.21.15.163:6800/1606811720 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fb54c0fb010 con 0x7fb53802fd10 2023-11-15T04:54:53.808 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: value unchanged 2023-11-15T04:54:53.810 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.809+0000 7fb551239700 1 -- 172.21.15.163:0/1376640604 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fb53802fd10 msgr2=0x7fb5380321d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:53.810 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.809+0000 7fb551239700 1 --2- 172.21.15.163:0/1376640604 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fb53802fd10 0x7fb5380321d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb5400096a0 tx=0x7fb540006b40).stop 2023-11-15T04:54:53.810 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.809+0000 7fb551239700 1 -- 172.21.15.163:0/1376640604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb54c0703f0 msgr2=0x7fb54c0fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:53.810 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.809+0000 7fb551239700 1 --2- 172.21.15.163:0/1376640604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb54c0703f0 0x7fb54c0fc3a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fb53c00e040 tx=0x7fb53c006ee0).stop 2023-11-15T04:54:53.811 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.809+0000 7fb551239700 1 -- 172.21.15.163:0/1376640604 shutdown_connections 2023-11-15T04:54:53.811 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.809+0000 7fb551239700 1 --2- 172.21.15.163:0/1376640604 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fb53802fd10 0x7fb5380321d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:53.811 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.809+0000 7fb551239700 1 --2- 172.21.15.163:0/1376640604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb54c0703f0 0x7fb54c0fc3a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:53.811 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.809+0000 7fb551239700 1 -- 172.21.15.163:0/1376640604 >> 172.21.15.163:0/1376640604 conn(0x7fb54c0f63c0 msgr2=0x7fb54c0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:53.811 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.809+0000 7fb551239700 1 -- 172.21.15.163:0/1376640604 shutdown_connections 2023-11-15T04:54:53.812 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:53.809+0000 7fb551239700 1 -- 172.21.15.163:0/1376640604 wait complete. 2023-11-15T04:54:54.121 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:54 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:52.907824+0000 mgr.smithi163.qesyrx (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:54:54.121 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:54 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:54:53.912318+0000 mon.smithi163 (mon.0) 50 : cluster [DBG] mgrmap e7: smithi163.qesyrx(active, since 2s) 2023-11-15T04:54:54.178 INFO:teuthology.orchestra.run.smithi163.stderr:Generating ssh key... 2023-11-15T04:54:54.178 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2023-11-15T04:54:54.523 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.521+0000 7ff6c35de700 1 Processor -- start 2023-11-15T04:54:54.523 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.521+0000 7ff6c35de700 1 -- start start 2023-11-15T04:54:54.524 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.521+0000 7ff6c35de700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff6bc0703b0 0x7ff6bc100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:54.524 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.521+0000 7ff6c35de700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6bc0fc750 con 0x7ff6bc0703b0 2023-11-15T04:54:54.524 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.522+0000 7ff6c137a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff6bc0703b0 0x7ff6bc100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:54.524 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.522+0000 7ff6c137a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff6bc0703b0 0x7ff6bc100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34912/0 (socket says 172.21.15.163:34912) 2023-11-15T04:54:54.524 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.522+0000 7ff6c137a700 1 -- 172.21.15.163:0/3109664262 learned_addr learned my addr 172.21.15.163:0/3109664262 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:54.525 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.522+0000 7ff6c137a700 1 -- 172.21.15.163:0/3109664262 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6bc101250 con 0x7ff6bc0703b0 2023-11-15T04:54:54.525 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.522+0000 7ff6c137a700 1 --2- 172.21.15.163:0/3109664262 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff6bc0703b0 0x7ff6bc100d60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ff6ac00b5f0 tx=0x7ff6ac006ee0).ready entity=mon.0 client_cookie=89e3e0cf77eb8377 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:54.525 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.522+0000 7ff6abfff700 1 -- 172.21.15.163:0/3109664262 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6ac00d600 con 0x7ff6bc0703b0 2023-11-15T04:54:54.525 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.522+0000 7ff6abfff700 1 -- 172.21.15.163:0/3109664262 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff6ac00dbd0 con 0x7ff6bc0703b0 2023-11-15T04:54:54.525 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.522+0000 7ff6abfff700 1 -- 172.21.15.163:0/3109664262 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6ac011c00 con 0x7ff6bc0703b0 2023-11-15T04:54:54.526 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.523+0000 7ff6c35de700 1 -- 172.21.15.163:0/3109664262 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff6bc0703b0 msgr2=0x7ff6bc100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:54.526 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.523+0000 7ff6c35de700 1 --2- 172.21.15.163:0/3109664262 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff6bc0703b0 0x7ff6bc100d60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ff6ac00b5f0 tx=0x7ff6ac006ee0).stop 2023-11-15T04:54:54.526 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.523+0000 7ff6c35de700 1 -- 172.21.15.163:0/3109664262 shutdown_connections 2023-11-15T04:54:54.526 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.523+0000 7ff6c35de700 1 --2- 172.21.15.163:0/3109664262 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff6bc0703b0 0x7ff6bc100d60 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:54.526 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.523+0000 7ff6c35de700 1 -- 172.21.15.163:0/3109664262 >> 172.21.15.163:0/3109664262 conn(0x7ff6bc0f63c0 msgr2=0x7ff6bc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:54.529 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.523+0000 7ff6c35de700 1 -- 172.21.15.163:0/3109664262 shutdown_connections 2023-11-15T04:54:54.529 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.523+0000 7ff6c35de700 1 -- 172.21.15.163:0/3109664262 wait complete. 2023-11-15T04:54:54.529 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.524+0000 7ff6c35de700 1 Processor -- start 2023-11-15T04:54:54.529 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.524+0000 7ff6c35de700 1 -- start start 2023-11-15T04:54:54.529 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.524+0000 7ff6c35de700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff6bc0703b0 0x7ff6bc10cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:54.530 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.524+0000 7ff6c35de700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6bc10d100 con 0x7ff6bc0703b0 2023-11-15T04:54:54.530 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.524+0000 7ff6c137a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff6bc0703b0 0x7ff6bc10cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:54.530 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.524+0000 7ff6c137a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff6bc0703b0 0x7ff6bc10cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34922/0 (socket says 172.21.15.163:34922) 2023-11-15T04:54:54.530 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.524+0000 7ff6c137a700 1 -- 172.21.15.163:0/205470330 learned_addr learned my addr 172.21.15.163:0/205470330 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:54.531 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.524+0000 7ff6c137a700 1 -- 172.21.15.163:0/205470330 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ac00b200 con 0x7ff6bc0703b0 2023-11-15T04:54:54.531 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.524+0000 7ff6c137a700 1 --2- 172.21.15.163:0/205470330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff6bc0703b0 0x7ff6bc10cc10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7ff6ac00b190 tx=0x7ff6ac006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:54.531 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.525+0000 7ff6a9ffb700 1 -- 172.21.15.163:0/205470330 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6ac01b410 con 0x7ff6bc0703b0 2023-11-15T04:54:54.531 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.525+0000 7ff6a9ffb700 1 -- 172.21.15.163:0/205470330 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff6ac01b9e0 con 0x7ff6bc0703b0 2023-11-15T04:54:54.531 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.525+0000 7ff6c35de700 1 -- 172.21.15.163:0/205470330 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6bc10d300 con 0x7ff6bc0703b0 2023-11-15T04:54:54.532 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.525+0000 7ff6a9ffb700 1 -- 172.21.15.163:0/205470330 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6ac01a880 con 0x7ff6bc0703b0 2023-11-15T04:54:54.532 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.525+0000 7ff6c35de700 1 -- 172.21.15.163:0/205470330 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6bc10d710 con 0x7ff6bc0703b0 2023-11-15T04:54:54.532 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.526+0000 7ff6a9ffb700 1 -- 172.21.15.163:0/205470330 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff6ac00f070 con 0x7ff6bc0703b0 2023-11-15T04:54:54.532 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.526+0000 7ff6a9ffb700 1 --2- 172.21.15.163:0/205470330 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7ff6a402fdb0 0x7ff6a4032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:54.532 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.526+0000 7ff6a9ffb700 1 -- 172.21.15.163:0/205470330 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ff6ac041220 con 0x7ff6bc0703b0 2023-11-15T04:54:54.532 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.526+0000 7ff6c35de700 1 -- 172.21.15.163:0/205470330 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6bc045ad0 con 0x7ff6bc0703b0 2023-11-15T04:54:54.533 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.526+0000 7ff6c0b79700 1 --2- 172.21.15.163:0/205470330 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7ff6a402fdb0 0x7ff6a4032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:54.533 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.526+0000 7ff6c0b79700 1 --2- 172.21.15.163:0/205470330 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7ff6a402fdb0 0x7ff6a4032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ff6b80096a0 tx=0x7ff6b8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:54.533 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.530+0000 7ff6a9ffb700 1 -- 172.21.15.163:0/205470330 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff6ac025460 con 0x7ff6bc0703b0 2023-11-15T04:54:54.681 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.681+0000 7ff6c35de700 1 -- 172.21.15.163:0/205470330 --> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6bc106a70 con 0x7ff6a402fdb0 2023-11-15T04:54:54.792 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.792+0000 7ff6a9ffb700 1 -- 172.21.15.163:0/205470330 <== mgr.14114 v2:172.21.15.163:6800/1606811720 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff6bc106a70 con 0x7ff6a402fdb0 2023-11-15T04:54:54.795 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.794+0000 7ff6c35de700 1 -- 172.21.15.163:0/205470330 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7ff6a402fdb0 msgr2=0x7ff6a4032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:54.795 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.794+0000 7ff6c35de700 1 --2- 172.21.15.163:0/205470330 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7ff6a402fdb0 0x7ff6a4032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ff6b80096a0 tx=0x7ff6b8006b40).stop 2023-11-15T04:54:54.795 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.794+0000 7ff6c35de700 1 -- 172.21.15.163:0/205470330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff6bc0703b0 msgr2=0x7ff6bc10cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:54.795 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.794+0000 7ff6c35de700 1 --2- 172.21.15.163:0/205470330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff6bc0703b0 0x7ff6bc10cc10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7ff6ac00b190 tx=0x7ff6ac006ee0).stop 2023-11-15T04:54:54.796 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.794+0000 7ff6c35de700 1 -- 172.21.15.163:0/205470330 shutdown_connections 2023-11-15T04:54:54.796 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.794+0000 7ff6c35de700 1 --2- 172.21.15.163:0/205470330 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7ff6a402fdb0 0x7ff6a4032270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:54.796 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.794+0000 7ff6c35de700 1 --2- 172.21.15.163:0/205470330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff6bc0703b0 0x7ff6bc10cc10 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:54.796 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.794+0000 7ff6c35de700 1 -- 172.21.15.163:0/205470330 >> 172.21.15.163:0/205470330 conn(0x7ff6bc0f63c0 msgr2=0x7ff6bc0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:54.796 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.794+0000 7ff6c35de700 1 -- 172.21.15.163:0/205470330 shutdown_connections 2023-11-15T04:54:54.797 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:54.794+0000 7ff6c35de700 1 -- 172.21.15.163:0/205470330 wait complete. 2023-11-15T04:54:55.169 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2023-11-15T04:54:55.318 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:55 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:53.807902+0000 mgr.smithi163.qesyrx (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:54:55.318 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:55 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:54.790145+0000 mon.smithi163 (mon.0) 51 : audit [INF] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' 2023-11-15T04:54:55.318 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:55 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:54.791887+0000 mon.smithi163 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' 2023-11-15T04:54:55.504 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.500+0000 7f61b3678700 1 Processor -- start 2023-11-15T04:54:55.504 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.500+0000 7f61b3678700 1 -- start start 2023-11-15T04:54:55.505 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.500+0000 7f61b3678700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f61ac0704a0 0x7f61ac0fe8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:55.505 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.500+0000 7f61b3678700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac0fede0 con 0x7f61ac0704a0 2023-11-15T04:54:55.505 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.500+0000 7f61b1414700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f61ac0704a0 0x7f61ac0fe8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:55.505 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.501+0000 7f61b1414700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f61ac0704a0 0x7f61ac0fe8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34926/0 (socket says 172.21.15.163:34926) 2023-11-15T04:54:55.505 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.501+0000 7f61b1414700 1 -- 172.21.15.163:0/2337979536 learned_addr learned my addr 172.21.15.163:0/2337979536 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:55.506 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.501+0000 7f61b1414700 1 -- 172.21.15.163:0/2337979536 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ac0fef20 con 0x7f61ac0704a0 2023-11-15T04:54:55.506 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.501+0000 7f61b1414700 1 --2- 172.21.15.163:0/2337979536 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f61ac0704a0 0x7f61ac0fe8f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f619c00b5f0 tx=0x7f619c006ee0).ready entity=mon.0 client_cookie=f80b3558ef8dc74b server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:55.506 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.501+0000 7f619bfff700 1 -- 172.21.15.163:0/2337979536 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f619c00d600 con 0x7f61ac0704a0 2023-11-15T04:54:55.507 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.501+0000 7f619bfff700 1 -- 172.21.15.163:0/2337979536 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f619c00dbd0 con 0x7f61ac0704a0 2023-11-15T04:54:55.507 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.501+0000 7f619bfff700 1 -- 172.21.15.163:0/2337979536 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f619c011c00 con 0x7f61ac0704a0 2023-11-15T04:54:55.507 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.502+0000 7f61b3678700 1 -- 172.21.15.163:0/2337979536 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f61ac0704a0 msgr2=0x7f61ac0fe8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:55.508 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.502+0000 7f61b3678700 1 --2- 172.21.15.163:0/2337979536 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f61ac0704a0 0x7f61ac0fe8f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f619c00b5f0 tx=0x7f619c006ee0).stop 2023-11-15T04:54:55.508 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.502+0000 7f61b3678700 1 -- 172.21.15.163:0/2337979536 shutdown_connections 2023-11-15T04:54:55.508 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.502+0000 7f61b3678700 1 --2- 172.21.15.163:0/2337979536 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f61ac0704a0 0x7f61ac0fe8f0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:55.508 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.502+0000 7f61b3678700 1 -- 172.21.15.163:0/2337979536 >> 172.21.15.163:0/2337979536 conn(0x7f61ac0f63b0 msgr2=0x7f61ac0f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:55.509 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.502+0000 7f61b3678700 1 -- 172.21.15.163:0/2337979536 shutdown_connections 2023-11-15T04:54:55.509 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.502+0000 7f61b3678700 1 -- 172.21.15.163:0/2337979536 wait complete. 2023-11-15T04:54:55.509 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.502+0000 7f61b3678700 1 Processor -- start 2023-11-15T04:54:55.509 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.502+0000 7f61b3678700 1 -- start start 2023-11-15T04:54:55.509 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.502+0000 7f61b3678700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f61ac0704a0 0x7f61ac0fc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:55.510 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.502+0000 7f61b3678700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac0fa7f0 con 0x7f61ac0704a0 2023-11-15T04:54:55.510 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.503+0000 7f61b1414700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f61ac0704a0 0x7f61ac0fc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:55.510 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.503+0000 7f61b1414700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f61ac0704a0 0x7f61ac0fc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34936/0 (socket says 172.21.15.163:34936) 2023-11-15T04:54:55.510 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.503+0000 7f61b1414700 1 -- 172.21.15.163:0/1418339129 learned_addr learned my addr 172.21.15.163:0/1418339129 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:55.511 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.503+0000 7f61b1414700 1 -- 172.21.15.163:0/1418339129 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619c00b200 con 0x7f61ac0704a0 2023-11-15T04:54:55.511 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.503+0000 7f61b1414700 1 --2- 172.21.15.163:0/1418339129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f61ac0704a0 0x7f61ac0fc370 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f619c00e040 tx=0x7f619c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:55.511 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.503+0000 7f6199ffb700 1 -- 172.21.15.163:0/1418339129 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f619c00d9f0 con 0x7f61ac0704a0 2023-11-15T04:54:55.511 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.503+0000 7f6199ffb700 1 -- 172.21.15.163:0/1418339129 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f619c01a9f0 con 0x7f61ac0704a0 2023-11-15T04:54:55.511 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.503+0000 7f6199ffb700 1 -- 172.21.15.163:0/1418339129 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f619c01b8c0 con 0x7f61ac0704a0 2023-11-15T04:54:55.511 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.503+0000 7f61b3678700 1 -- 172.21.15.163:0/1418339129 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61ac0fa990 con 0x7f61ac0704a0 2023-11-15T04:54:55.512 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.503+0000 7f61b3678700 1 -- 172.21.15.163:0/1418339129 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61ac0fada0 con 0x7f61ac0704a0 2023-11-15T04:54:55.513 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.504+0000 7f61b3678700 1 -- 172.21.15.163:0/1418339129 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61ac045ad0 con 0x7f61ac0704a0 2023-11-15T04:54:55.514 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.505+0000 7f6199ffb700 1 -- 172.21.15.163:0/1418339129 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f619c00f070 con 0x7f61ac0704a0 2023-11-15T04:54:55.514 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.505+0000 7f6199ffb700 1 --2- 172.21.15.163:0/1418339129 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f619402fdb0 0x7f6194032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:55.514 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.505+0000 7f6199ffb700 1 -- 172.21.15.163:0/1418339129 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f619c041980 con 0x7f61ac0704a0 2023-11-15T04:54:55.514 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.505+0000 7f61b0c13700 1 --2- 172.21.15.163:0/1418339129 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f619402fdb0 0x7f6194032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:55.515 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.505+0000 7f61b0c13700 1 --2- 172.21.15.163:0/1418339129 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f619402fdb0 0x7f6194032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f61a80096a0 tx=0x7f61a8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:55.515 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.509+0000 7f6199ffb700 1 -- 172.21.15.163:0/1418339129 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f619c02d7a0 con 0x7f61ac0704a0 2023-11-15T04:54:55.661 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.660+0000 7f61b3678700 1 -- 172.21.15.163:0/1418339129 --> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f61ac0fafe0 con 0x7f619402fdb0 2023-11-15T04:54:55.661 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.660+0000 7f6199ffb700 1 -- 172.21.15.163:0/1418339129 <== mgr.14114 v2:172.21.15.163:6800/1606811720 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f61ac0fafe0 con 0x7f619402fdb0 2023-11-15T04:54:55.661 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCh0oJUIrrpSBox3sxjwxB0vqh1Jf8sMaw/tuitAfQKRUHPsNamy3MCuZ5W1n7nDoBNSPQB+qd1hL1AkZJg3dnHL6OHwwIWTkKWHOBZdYIOZhu9XJtdEWb2aHQ5Pjm5ikby+r7zLBvtnmvpCn06z1BDVIa6sozeajmkVjcWlZZZ1JVeasmNdRA30CZFTicm16qxGwPZYycFmGmgL1bxYRTptOZISQygCZKPoDzIJrON/tw6sQVbuh/vp0fArlGbGoW2enqXCDb9nvSIjSqU1L2EbA1Kp5bn0i3yn5BrQ6qe+C16MPTT3YFngYtazjExYCdefardidOHHhzl0EkuobA9DkAI017WA0kvVVfpAhMc/1rh86zdptf5pb5SEEz8pQoYLhgysliqs5GhSu+0iGR1aH3N2ifE3ryybGFx+15ObJVx58sozwjVTzWMs2sCeMywQ2jpRxux2YUQ1VH0qv3WFQt9PD3ob/Xdm25FDh6nNf4kLP3EWG9+guGCpy4aKs= ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7 2023-11-15T04:54:55.664 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.662+0000 7f61b3678700 1 -- 172.21.15.163:0/1418339129 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f619402fdb0 msgr2=0x7f6194032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:55.664 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.662+0000 7f61b3678700 1 --2- 172.21.15.163:0/1418339129 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f619402fdb0 0x7f6194032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f61a80096a0 tx=0x7f61a8006b40).stop 2023-11-15T04:54:55.664 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.662+0000 7f61b3678700 1 -- 172.21.15.163:0/1418339129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f61ac0704a0 msgr2=0x7f61ac0fc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:55.665 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.662+0000 7f61b3678700 1 --2- 172.21.15.163:0/1418339129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f61ac0704a0 0x7f61ac0fc370 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f619c00e040 tx=0x7f619c006ee0).stop 2023-11-15T04:54:55.665 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.662+0000 7f61b3678700 1 -- 172.21.15.163:0/1418339129 shutdown_connections 2023-11-15T04:54:55.665 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.662+0000 7f61b3678700 1 --2- 172.21.15.163:0/1418339129 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f619402fdb0 0x7f6194032270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:55.665 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.662+0000 7f61b3678700 1 --2- 172.21.15.163:0/1418339129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f61ac0704a0 0x7f61ac0fc370 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:55.665 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.662+0000 7f61b3678700 1 -- 172.21.15.163:0/1418339129 >> 172.21.15.163:0/1418339129 conn(0x7f61ac0f63b0 msgr2=0x7f61ac0f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:55.666 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.662+0000 7f61b3678700 1 -- 172.21.15.163:0/1418339129 shutdown_connections 2023-11-15T04:54:55.666 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:55.662+0000 7f61b3678700 1 -- 172.21.15.163:0/1418339129 wait complete. 2023-11-15T04:54:56.002 INFO:teuthology.orchestra.run.smithi163.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-11-15T04:54:56.002 INFO:teuthology.orchestra.run.smithi163.stderr:Adding key to root@localhost authorized_keys... 2023-11-15T04:54:56.003 INFO:teuthology.orchestra.run.smithi163.stderr:Adding host smithi163... 2023-11-15T04:54:56.003 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi163 172.21.15.163 2023-11-15T04:54:56.364 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.362+0000 7fbd74a5f700 1 Processor -- start 2023-11-15T04:54:56.365 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.362+0000 7fbd74a5f700 1 -- start start 2023-11-15T04:54:56.365 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.362+0000 7fbd74a5f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd700703d0 0x7fbd70100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:56.365 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.362+0000 7fbd74a5f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd700fc730 con 0x7fbd700703d0 2023-11-15T04:54:56.365 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.362+0000 7fbd6e59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd700703d0 0x7fbd70100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:56.366 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.363+0000 7fbd6e59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd700703d0 0x7fbd70100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34952/0 (socket says 172.21.15.163:34952) 2023-11-15T04:54:56.366 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.363+0000 7fbd6e59c700 1 -- 172.21.15.163:0/3340144818 learned_addr learned my addr 172.21.15.163:0/3340144818 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:56.366 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.363+0000 7fbd6e59c700 1 -- 172.21.15.163:0/3340144818 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd70101230 con 0x7fbd700703d0 2023-11-15T04:54:56.366 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.363+0000 7fbd6e59c700 1 --2- 172.21.15.163:0/3340144818 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd700703d0 0x7fbd70100d40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fbd6400b5f0 tx=0x7fbd64006ee0).ready entity=mon.0 client_cookie=645a93e85be0283e server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:56.366 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.363+0000 7fbd6d59a700 1 -- 172.21.15.163:0/3340144818 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd6400d600 con 0x7fbd700703d0 2023-11-15T04:54:56.367 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.363+0000 7fbd6d59a700 1 -- 172.21.15.163:0/3340144818 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbd6400dbd0 con 0x7fbd700703d0 2023-11-15T04:54:56.367 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.363+0000 7fbd6d59a700 1 -- 172.21.15.163:0/3340144818 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd64011c00 con 0x7fbd700703d0 2023-11-15T04:54:56.367 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.364+0000 7fbd74a5f700 1 -- 172.21.15.163:0/3340144818 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd700703d0 msgr2=0x7fbd70100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:56.367 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.364+0000 7fbd74a5f700 1 --2- 172.21.15.163:0/3340144818 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd700703d0 0x7fbd70100d40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fbd6400b5f0 tx=0x7fbd64006ee0).stop 2023-11-15T04:54:56.367 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.364+0000 7fbd74a5f700 1 -- 172.21.15.163:0/3340144818 shutdown_connections 2023-11-15T04:54:56.368 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.364+0000 7fbd74a5f700 1 --2- 172.21.15.163:0/3340144818 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd700703d0 0x7fbd70100d40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:56.368 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.364+0000 7fbd74a5f700 1 -- 172.21.15.163:0/3340144818 >> 172.21.15.163:0/3340144818 conn(0x7fbd700f63c0 msgr2=0x7fbd700f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:56.368 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.364+0000 7fbd74a5f700 1 -- 172.21.15.163:0/3340144818 shutdown_connections 2023-11-15T04:54:56.368 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.364+0000 7fbd74a5f700 1 -- 172.21.15.163:0/3340144818 wait complete. 2023-11-15T04:54:56.370 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.364+0000 7fbd74a5f700 1 Processor -- start 2023-11-15T04:54:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.364+0000 7fbd74a5f700 1 -- start start 2023-11-15T04:54:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.365+0000 7fbd74a5f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd700703d0 0x7fbd7010cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.365+0000 7fbd74a5f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7010d100 con 0x7fbd700703d0 2023-11-15T04:54:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.365+0000 7fbd6e59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd700703d0 0x7fbd7010cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:56.372 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.365+0000 7fbd6e59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd700703d0 0x7fbd7010cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34958/0 (socket says 172.21.15.163:34958) 2023-11-15T04:54:56.372 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.365+0000 7fbd6e59c700 1 -- 172.21.15.163:0/2825437267 learned_addr learned my addr 172.21.15.163:0/2825437267 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:54:56.372 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.365+0000 7fbd6e59c700 1 -- 172.21.15.163:0/2825437267 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd6400b200 con 0x7fbd700703d0 2023-11-15T04:54:56.372 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.365+0000 7fbd6e59c700 1 --2- 172.21.15.163:0/2825437267 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd700703d0 0x7fbd7010cc10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fbd6400e040 tx=0x7fbd64006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:56.372 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.365+0000 7fbd5effd700 1 -- 172.21.15.163:0/2825437267 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd6400d9f0 con 0x7fbd700703d0 2023-11-15T04:54:56.373 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.365+0000 7fbd5effd700 1 -- 172.21.15.163:0/2825437267 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbd6401a9f0 con 0x7fbd700703d0 2023-11-15T04:54:56.373 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.365+0000 7fbd5effd700 1 -- 172.21.15.163:0/2825437267 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd6401b8c0 con 0x7fbd700703d0 2023-11-15T04:54:56.373 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.365+0000 7fbd74a5f700 1 -- 172.21.15.163:0/2825437267 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd7010d300 con 0x7fbd700703d0 2023-11-15T04:54:56.373 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.365+0000 7fbd74a5f700 1 -- 172.21.15.163:0/2825437267 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd7010d710 con 0x7fbd700703d0 2023-11-15T04:54:56.373 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.366+0000 7fbd5effd700 1 -- 172.21.15.163:0/2825437267 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fbd6400f070 con 0x7fbd700703d0 2023-11-15T04:54:56.374 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.366+0000 7fbd5effd700 1 --2- 172.21.15.163:0/2825437267 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fbd5402fdb0 0x7fbd54032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:54:56.374 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.366+0000 7fbd5effd700 1 -- 172.21.15.163:0/2825437267 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fbd640412a0 con 0x7fbd700703d0 2023-11-15T04:54:56.374 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.366+0000 7fbd6dd9b700 1 --2- 172.21.15.163:0/2825437267 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fbd5402fdb0 0x7fbd54032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:54:56.374 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.367+0000 7fbd6dd9b700 1 --2- 172.21.15.163:0/2825437267 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fbd5402fdb0 0x7fbd54032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fbd600096a0 tx=0x7fbd60006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:54:56.374 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.367+0000 7fbd74a5f700 1 -- 172.21.15.163:0/2825437267 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd70059100 con 0x7fbd700703d0 2023-11-15T04:54:56.375 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.371+0000 7fbd5effd700 1 -- 172.21.15.163:0/2825437267 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd64016070 con 0x7fbd700703d0 2023-11-15T04:54:56.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:56 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cephadm 2023-11-15T04:54:54.682459+0000 mgr.smithi163.qesyrx (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2023-11-15T04:54:56.540 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:56.540+0000 7fbd74a5f700 1 -- 172.21.15.163:0/2825437267 --> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi163", "addr": "172.21.15.163", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd70106a90 con 0x7fbd5402fdb0 2023-11-15T04:54:57.221 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: Added host 'smithi163' 2023-11-15T04:54:57.222 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:57.219+0000 7fbd5effd700 1 -- 172.21.15.163:0/2825437267 <== mgr.14114 v2:172.21.15.163:6800/1606811720 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fbd70106a90 con 0x7fbd5402fdb0 2023-11-15T04:54:57.222 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:57.222+0000 7fbd74a5f700 1 -- 172.21.15.163:0/2825437267 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fbd5402fdb0 msgr2=0x7fbd54032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:57.222 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:57.222+0000 7fbd74a5f700 1 --2- 172.21.15.163:0/2825437267 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fbd5402fdb0 0x7fbd54032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fbd600096a0 tx=0x7fbd60006b40).stop 2023-11-15T04:54:57.223 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:57.222+0000 7fbd74a5f700 1 -- 172.21.15.163:0/2825437267 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd700703d0 msgr2=0x7fbd7010cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:54:57.223 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:57.222+0000 7fbd74a5f700 1 --2- 172.21.15.163:0/2825437267 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd700703d0 0x7fbd7010cc10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fbd6400e040 tx=0x7fbd64006ee0).stop 2023-11-15T04:54:57.223 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:57.222+0000 7fbd74a5f700 1 -- 172.21.15.163:0/2825437267 shutdown_connections 2023-11-15T04:54:57.223 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:57.222+0000 7fbd74a5f700 1 --2- 172.21.15.163:0/2825437267 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fbd5402fdb0 0x7fbd54032270 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:57.224 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:57.222+0000 7fbd74a5f700 1 --2- 172.21.15.163:0/2825437267 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd700703d0 0x7fbd7010cc10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:54:57.224 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:57.222+0000 7fbd74a5f700 1 -- 172.21.15.163:0/2825437267 >> 172.21.15.163:0/2825437267 conn(0x7fbd700f63c0 msgr2=0x7fbd700f7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:54:57.224 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:57.222+0000 7fbd74a5f700 1 -- 172.21.15.163:0/2825437267 shutdown_connections 2023-11-15T04:54:57.224 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:54:57.222+0000 7fbd74a5f700 1 -- 172.21.15.163:0/2825437267 wait complete. 2023-11-15T04:54:57.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:57 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:55.661181+0000 mgr.smithi163.qesyrx (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:54:58.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:58 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:56.541525+0000 mgr.smithi163.qesyrx (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi163", "addr": "172.21.15.163", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:54:58.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:58 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:57.219160+0000 mon.smithi163 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' 2023-11-15T04:54:58.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:58 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:54:57.219710+0000 mon.smithi163 (mon.0) 54 : audit [DBG] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:54:58.844 INFO:teuthology.orchestra.run.smithi163.stderr:Deploying mon service with default placement... 2023-11-15T04:54:58.844 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2023-11-15T04:54:59.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:54:59 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cephadm 2023-11-15T04:54:57.219480+0000 mgr.smithi163.qesyrx (mgr.14114) 9 : cephadm [INF] Added host smithi163 2023-11-15T04:55:00.479 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.270+0000 7fb6682e5700 1 Processor -- start 2023-11-15T04:55:00.479 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.270+0000 7fb6682e5700 1 -- start start 2023-11-15T04:55:00.480 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.270+0000 7fb6682e5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6600fab80 0x7fb6600fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:00.480 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.270+0000 7fb6682e5700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6600fb490 con 0x7fb6600fab80 2023-11-15T04:55:00.480 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.271+0000 7fb666081700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6600fab80 0x7fb6600fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:00.480 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.271+0000 7fb666081700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6600fab80 0x7fb6600fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34970/0 (socket says 172.21.15.163:34970) 2023-11-15T04:55:00.481 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.271+0000 7fb666081700 1 -- 172.21.15.163:0/380583093 learned_addr learned my addr 172.21.15.163:0/380583093 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:00.481 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.271+0000 7fb666081700 1 -- 172.21.15.163:0/380583093 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6600fb5d0 con 0x7fb6600fab80 2023-11-15T04:55:00.481 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.271+0000 7fb666081700 1 --2- 172.21.15.163:0/380583093 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6600fab80 0x7fb6600fafa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fb65c00b870 tx=0x7fb65c006f70).ready entity=mon.0 client_cookie=5627493b01d5bb06 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:00.481 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.271+0000 7fb66507f700 1 -- 172.21.15.163:0/380583093 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb65c00da20 con 0x7fb6600fab80 2023-11-15T04:55:00.481 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.271+0000 7fb66507f700 1 -- 172.21.15.163:0/380583093 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb65c010470 con 0x7fb6600fab80 2023-11-15T04:55:00.482 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.271+0000 7fb66507f700 1 -- 172.21.15.163:0/380583093 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb65c00f5b0 con 0x7fb6600fab80 2023-11-15T04:55:00.482 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.272+0000 7fb6682e5700 1 -- 172.21.15.163:0/380583093 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6600fab80 msgr2=0x7fb6600fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:00.482 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.272+0000 7fb6682e5700 1 --2- 172.21.15.163:0/380583093 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6600fab80 0x7fb6600fafa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fb65c00b870 tx=0x7fb65c006f70).stop 2023-11-15T04:55:00.482 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.272+0000 7fb6682e5700 1 -- 172.21.15.163:0/380583093 shutdown_connections 2023-11-15T04:55:00.482 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.272+0000 7fb6682e5700 1 --2- 172.21.15.163:0/380583093 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6600fab80 0x7fb6600fafa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:00.483 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.272+0000 7fb6682e5700 1 -- 172.21.15.163:0/380583093 >> 172.21.15.163:0/380583093 conn(0x7fb6600f63c0 msgr2=0x7fb6600f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:00.483 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.272+0000 7fb6682e5700 1 -- 172.21.15.163:0/380583093 shutdown_connections 2023-11-15T04:55:00.483 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.272+0000 7fb6682e5700 1 -- 172.21.15.163:0/380583093 wait complete. 2023-11-15T04:55:00.483 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.272+0000 7fb6682e5700 1 Processor -- start 2023-11-15T04:55:00.483 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.272+0000 7fb6682e5700 1 -- start start 2023-11-15T04:55:00.484 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.272+0000 7fb6682e5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6600fab80 0x7fb66010cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:00.484 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.272+0000 7fb6682e5700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66010d070 con 0x7fb6600fab80 2023-11-15T04:55:00.484 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.273+0000 7fb666081700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6600fab80 0x7fb66010cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:00.484 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.273+0000 7fb666081700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6600fab80 0x7fb66010cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34976/0 (socket says 172.21.15.163:34976) 2023-11-15T04:55:00.484 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.273+0000 7fb666081700 1 -- 172.21.15.163:0/2355868274 learned_addr learned my addr 172.21.15.163:0/2355868274 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:00.485 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.273+0000 7fb666081700 1 -- 172.21.15.163:0/2355868274 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb65c00b480 con 0x7fb6600fab80 2023-11-15T04:55:00.485 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.273+0000 7fb666081700 1 --2- 172.21.15.163:0/2355868274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6600fab80 0x7fb66010cb80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fb65c006800 tx=0x7fb65c006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:00.485 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.273+0000 7fb64effd700 1 -- 172.21.15.163:0/2355868274 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb65c00f980 con 0x7fb6600fab80 2023-11-15T04:55:00.485 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.273+0000 7fb64effd700 1 -- 172.21.15.163:0/2355868274 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb65c019450 con 0x7fb6600fab80 2023-11-15T04:55:00.485 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.273+0000 7fb64effd700 1 -- 172.21.15.163:0/2355868274 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb65c023640 con 0x7fb6600fab80 2023-11-15T04:55:00.485 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.273+0000 7fb6682e5700 1 -- 172.21.15.163:0/2355868274 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb66010d270 con 0x7fb6600fab80 2023-11-15T04:55:00.486 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.273+0000 7fb6682e5700 1 -- 172.21.15.163:0/2355868274 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb66010d680 con 0x7fb6600fab80 2023-11-15T04:55:00.486 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.274+0000 7fb64effd700 1 -- 172.21.15.163:0/2355868274 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb65c016030 con 0x7fb6600fab80 2023-11-15T04:55:00.486 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.274+0000 7fb64effd700 1 --2- 172.21.15.163:0/2355868274 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fb64802fdb0 0x7fb648032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:00.486 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.274+0000 7fb64effd700 1 -- 172.21.15.163:0/2355868274 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb65c041970 con 0x7fb6600fab80 2023-11-15T04:55:00.486 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.274+0000 7fb6682e5700 1 -- 172.21.15.163:0/2355868274 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb660045ad0 con 0x7fb6600fab80 2023-11-15T04:55:00.487 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.274+0000 7fb665880700 1 --2- 172.21.15.163:0/2355868274 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fb64802fdb0 0x7fb648032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:00.487 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.275+0000 7fb665880700 1 --2- 172.21.15.163:0/2355868274 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fb64802fdb0 0x7fb648032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fb6500096a0 tx=0x7fb650006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:00.487 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.279+0000 7fb64effd700 1 -- 172.21.15.163:0/2355868274 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb65c014080 con 0x7fb6600fab80 2023-11-15T04:55:00.487 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.455+0000 7fb6682e5700 1 -- 172.21.15.163:0/2355868274 --> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6601026b0 con 0x7fb64802fdb0 2023-11-15T04:55:00.487 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.459+0000 7fb64effd700 1 -- 172.21.15.163:0/2355868274 <== mgr.14114 v2:172.21.15.163:6800/1606811720 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb6601026b0 con 0x7fb64802fdb0 2023-11-15T04:55:00.488 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.460+0000 7fb6682e5700 1 -- 172.21.15.163:0/2355868274 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fb64802fdb0 msgr2=0x7fb648032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:00.488 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.460+0000 7fb6682e5700 1 --2- 172.21.15.163:0/2355868274 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fb64802fdb0 0x7fb648032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fb6500096a0 tx=0x7fb650006b40).stop 2023-11-15T04:55:00.488 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.460+0000 7fb6682e5700 1 -- 172.21.15.163:0/2355868274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6600fab80 msgr2=0x7fb66010cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:00.488 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.460+0000 7fb6682e5700 1 --2- 172.21.15.163:0/2355868274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6600fab80 0x7fb66010cb80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fb65c006800 tx=0x7fb65c006f70).stop 2023-11-15T04:55:00.488 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.460+0000 7fb6682e5700 1 -- 172.21.15.163:0/2355868274 shutdown_connections 2023-11-15T04:55:00.489 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.460+0000 7fb6682e5700 1 --2- 172.21.15.163:0/2355868274 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fb64802fdb0 0x7fb648032270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:00.489 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.460+0000 7fb6682e5700 1 --2- 172.21.15.163:0/2355868274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6600fab80 0x7fb66010cb80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:00.489 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.461+0000 7fb6682e5700 1 -- 172.21.15.163:0/2355868274 >> 172.21.15.163:0/2355868274 conn(0x7fb6600f63c0 msgr2=0x7fb6600ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:00.489 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.461+0000 7fb6682e5700 1 -- 172.21.15.163:0/2355868274 shutdown_connections 2023-11-15T04:55:00.489 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:00.461+0000 7fb6682e5700 1 -- 172.21.15.163:0/2355868274 wait complete. 2023-11-15T04:55:00.489 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: Scheduled mon update... 2023-11-15T04:55:01.953 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:01 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:00.456135+0000 mgr.smithi163.qesyrx (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:55:01.953 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:01 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cephadm 2023-11-15T04:55:00.456836+0000 mgr.smithi163.qesyrx (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2023-11-15T04:55:01.954 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:01 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:00.459263+0000 mon.smithi163 (mon.0) 55 : audit [INF] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' 2023-11-15T04:55:02.317 INFO:teuthology.orchestra.run.smithi163.stderr:Deploying mgr service with default placement... 2023-11-15T04:55:02.318 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2023-11-15T04:55:03.551 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.514+0000 7fd861cec700 1 Processor -- start 2023-11-15T04:55:03.552 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.514+0000 7fd861cec700 1 -- start start 2023-11-15T04:55:03.552 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.515+0000 7fd861cec700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd85c0fab80 0x7fd85c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:03.552 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.515+0000 7fd861cec700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c0fb490 con 0x7fd85c0fab80 2023-11-15T04:55:03.552 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.515+0000 7fd85b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd85c0fab80 0x7fd85c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:03.552 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.515+0000 7fd85b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd85c0fab80 0x7fd85c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36858/0 (socket says 172.21.15.163:36858) 2023-11-15T04:55:03.553 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.515+0000 7fd85b7fe700 1 -- 172.21.15.163:0/1686181986 learned_addr learned my addr 172.21.15.163:0/1686181986 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:03.553 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.515+0000 7fd85b7fe700 1 -- 172.21.15.163:0/1686181986 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85c0fb5d0 con 0x7fd85c0fab80 2023-11-15T04:55:03.553 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.515+0000 7fd85b7fe700 1 --2- 172.21.15.163:0/1686181986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd85c0fab80 0x7fd85c0fafa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fd85000b5f0 tx=0x7fd850006ee0).ready entity=mon.0 client_cookie=4d53eb6dcbbcadb7 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:03.553 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.515+0000 7fd85a7fc700 1 -- 172.21.15.163:0/1686181986 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd85000d600 con 0x7fd85c0fab80 2023-11-15T04:55:03.553 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.515+0000 7fd85a7fc700 1 -- 172.21.15.163:0/1686181986 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd85000dbd0 con 0x7fd85c0fab80 2023-11-15T04:55:03.554 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.515+0000 7fd85a7fc700 1 -- 172.21.15.163:0/1686181986 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd850011c00 con 0x7fd85c0fab80 2023-11-15T04:55:03.554 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.516+0000 7fd861cec700 1 -- 172.21.15.163:0/1686181986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd85c0fab80 msgr2=0x7fd85c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:03.554 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.516+0000 7fd861cec700 1 --2- 172.21.15.163:0/1686181986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd85c0fab80 0x7fd85c0fafa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fd85000b5f0 tx=0x7fd850006ee0).stop 2023-11-15T04:55:03.554 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.516+0000 7fd861cec700 1 -- 172.21.15.163:0/1686181986 shutdown_connections 2023-11-15T04:55:03.554 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.516+0000 7fd861cec700 1 --2- 172.21.15.163:0/1686181986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd85c0fab80 0x7fd85c0fafa0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:03.555 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.516+0000 7fd861cec700 1 -- 172.21.15.163:0/1686181986 >> 172.21.15.163:0/1686181986 conn(0x7fd85c0f63c0 msgr2=0x7fd85c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:03.555 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.516+0000 7fd861cec700 1 -- 172.21.15.163:0/1686181986 shutdown_connections 2023-11-15T04:55:03.555 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.516+0000 7fd861cec700 1 -- 172.21.15.163:0/1686181986 wait complete. 2023-11-15T04:55:03.555 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.517+0000 7fd861cec700 1 Processor -- start 2023-11-15T04:55:03.555 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.517+0000 7fd861cec700 1 -- start start 2023-11-15T04:55:03.555 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.517+0000 7fd861cec700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd85c0fab80 0x7fd85c10fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:03.556 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.517+0000 7fd861cec700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c10b530 con 0x7fd85c0fab80 2023-11-15T04:55:03.556 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.517+0000 7fd85b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd85c0fab80 0x7fd85c10fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:03.556 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.517+0000 7fd85b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd85c0fab80 0x7fd85c10fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36874/0 (socket says 172.21.15.163:36874) 2023-11-15T04:55:03.557 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.517+0000 7fd85b7fe700 1 -- 172.21.15.163:0/2743107723 learned_addr learned my addr 172.21.15.163:0/2743107723 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:03.557 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.517+0000 7fd85b7fe700 1 -- 172.21.15.163:0/2743107723 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85000b200 con 0x7fd85c0fab80 2023-11-15T04:55:03.557 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.517+0000 7fd85b7fe700 1 --2- 172.21.15.163:0/2743107723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd85c0fab80 0x7fd85c10fb40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fd85000e040 tx=0x7fd850006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:03.557 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.518+0000 7fd843fff700 1 -- 172.21.15.163:0/2743107723 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd85001b410 con 0x7fd85c0fab80 2023-11-15T04:55:03.557 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.518+0000 7fd843fff700 1 -- 172.21.15.163:0/2743107723 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd85001b9e0 con 0x7fd85c0fab80 2023-11-15T04:55:03.558 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.518+0000 7fd861cec700 1 -- 172.21.15.163:0/2743107723 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd85c070450 con 0x7fd85c0fab80 2023-11-15T04:55:03.558 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.518+0000 7fd843fff700 1 -- 172.21.15.163:0/2743107723 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd85001a820 con 0x7fd85c0fab80 2023-11-15T04:55:03.558 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.518+0000 7fd861cec700 1 -- 172.21.15.163:0/2743107723 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd85c110030 con 0x7fd85c0fab80 2023-11-15T04:55:03.558 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.519+0000 7fd843fff700 1 -- 172.21.15.163:0/2743107723 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fd85000f070 con 0x7fd85c0fab80 2023-11-15T04:55:03.558 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.519+0000 7fd843fff700 1 --2- 172.21.15.163:0/2743107723 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fd84802fd50 0x7fd848032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:03.559 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.519+0000 7fd843fff700 1 -- 172.21.15.163:0/2743107723 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fd850041440 con 0x7fd85c0fab80 2023-11-15T04:55:03.559 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.519+0000 7fd861cec700 1 -- 172.21.15.163:0/2743107723 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd85c045ad0 con 0x7fd85c0fab80 2023-11-15T04:55:03.559 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.519+0000 7fd85affd700 1 --2- 172.21.15.163:0/2743107723 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fd84802fd50 0x7fd848032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:03.559 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.519+0000 7fd85affd700 1 --2- 172.21.15.163:0/2743107723 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fd84802fd50 0x7fd848032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fd84c0096a0 tx=0x7fd84c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:03.559 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.523+0000 7fd843fff700 1 -- 172.21.15.163:0/2743107723 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd850016070 con 0x7fd85c0fab80 2023-11-15T04:55:03.706 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.705+0000 7fd861cec700 1 -- 172.21.15.163:0/2743107723 --> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fd85c0fc960 con 0x7fd84802fd50 2023-11-15T04:55:03.709 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.709+0000 7fd843fff700 1 -- 172.21.15.163:0/2743107723 <== mgr.14114 v2:172.21.15.163:6800/1606811720 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd85c0fc960 con 0x7fd84802fd50 2023-11-15T04:55:03.709 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-11-15T04:55:03.712 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.711+0000 7fd861cec700 1 -- 172.21.15.163:0/2743107723 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fd84802fd50 msgr2=0x7fd848032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:03.712 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.711+0000 7fd861cec700 1 --2- 172.21.15.163:0/2743107723 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fd84802fd50 0x7fd848032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fd84c0096a0 tx=0x7fd84c006b40).stop 2023-11-15T04:55:03.712 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.711+0000 7fd861cec700 1 -- 172.21.15.163:0/2743107723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd85c0fab80 msgr2=0x7fd85c10fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:03.712 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.711+0000 7fd861cec700 1 --2- 172.21.15.163:0/2743107723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd85c0fab80 0x7fd85c10fb40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fd85000e040 tx=0x7fd850006ee0).stop 2023-11-15T04:55:03.713 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.711+0000 7fd861cec700 1 -- 172.21.15.163:0/2743107723 shutdown_connections 2023-11-15T04:55:03.713 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.711+0000 7fd861cec700 1 --2- 172.21.15.163:0/2743107723 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fd84802fd50 0x7fd848032210 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:03.713 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.711+0000 7fd861cec700 1 --2- 172.21.15.163:0/2743107723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd85c0fab80 0x7fd85c10fb40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:03.713 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.711+0000 7fd861cec700 1 -- 172.21.15.163:0/2743107723 >> 172.21.15.163:0/2743107723 conn(0x7fd85c0f63c0 msgr2=0x7fd85c0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:03.713 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.712+0000 7fd861cec700 1 -- 172.21.15.163:0/2743107723 shutdown_connections 2023-11-15T04:55:03.714 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:03.712+0000 7fd861cec700 1 -- 172.21.15.163:0/2743107723 wait complete. 2023-11-15T04:55:04.456 INFO:teuthology.orchestra.run.smithi163.stderr:Deploying crash service with default placement... 2023-11-15T04:55:04.456 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2023-11-15T04:55:04.806 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.799+0000 7fb57fdbe700 1 Processor -- start 2023-11-15T04:55:04.806 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.799+0000 7fb57fdbe700 1 -- start start 2023-11-15T04:55:04.806 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.799+0000 7fb57fdbe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb578070430 0x7fb5780fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:04.806 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.799+0000 7fb57fdbe700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5780fedd0 con 0x7fb578070430 2023-11-15T04:55:04.807 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.799+0000 7fb57db5a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb578070430 0x7fb5780fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:04.807 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.799+0000 7fb57db5a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb578070430 0x7fb5780fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36888/0 (socket says 172.21.15.163:36888) 2023-11-15T04:55:04.807 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.799+0000 7fb57db5a700 1 -- 172.21.15.163:0/3623709596 learned_addr learned my addr 172.21.15.163:0/3623709596 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:04.807 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.799+0000 7fb57db5a700 1 -- 172.21.15.163:0/3623709596 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5780fef10 con 0x7fb578070430 2023-11-15T04:55:04.808 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.800+0000 7fb57db5a700 1 --2- 172.21.15.163:0/3623709596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb578070430 0x7fb5780fe8e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb57400b5f0 tx=0x7fb574006ee0).ready entity=mon.0 client_cookie=5fdfd16fbee04351 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:04.808 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.800+0000 7fb57cb58700 1 -- 172.21.15.163:0/3623709596 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb57400d600 con 0x7fb578070430 2023-11-15T04:55:04.808 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.800+0000 7fb57cb58700 1 -- 172.21.15.163:0/3623709596 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb57400dbd0 con 0x7fb578070430 2023-11-15T04:55:04.808 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.800+0000 7fb57cb58700 1 -- 172.21.15.163:0/3623709596 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb574011c00 con 0x7fb578070430 2023-11-15T04:55:04.808 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.800+0000 7fb57fdbe700 1 -- 172.21.15.163:0/3623709596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb578070430 msgr2=0x7fb5780fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:04.809 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.800+0000 7fb57fdbe700 1 --2- 172.21.15.163:0/3623709596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb578070430 0x7fb5780fe8e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb57400b5f0 tx=0x7fb574006ee0).stop 2023-11-15T04:55:04.809 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.800+0000 7fb57fdbe700 1 -- 172.21.15.163:0/3623709596 shutdown_connections 2023-11-15T04:55:04.809 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.800+0000 7fb57fdbe700 1 --2- 172.21.15.163:0/3623709596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb578070430 0x7fb5780fe8e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:04.809 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.800+0000 7fb57fdbe700 1 -- 172.21.15.163:0/3623709596 >> 172.21.15.163:0/3623709596 conn(0x7fb5780f63c0 msgr2=0x7fb5780f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:04.810 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.800+0000 7fb57fdbe700 1 -- 172.21.15.163:0/3623709596 shutdown_connections 2023-11-15T04:55:04.810 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.800+0000 7fb57fdbe700 1 -- 172.21.15.163:0/3623709596 wait complete. 2023-11-15T04:55:04.810 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.801+0000 7fb57fdbe700 1 Processor -- start 2023-11-15T04:55:04.810 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.801+0000 7fb57fdbe700 1 -- start start 2023-11-15T04:55:04.810 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.801+0000 7fb57fdbe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb578070430 0x7fb57810aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:04.811 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.801+0000 7fb57fdbe700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57810aef0 con 0x7fb578070430 2023-11-15T04:55:04.811 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.801+0000 7fb57db5a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb578070430 0x7fb57810aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:04.811 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.801+0000 7fb57db5a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb578070430 0x7fb57810aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36902/0 (socket says 172.21.15.163:36902) 2023-11-15T04:55:04.812 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.801+0000 7fb57db5a700 1 -- 172.21.15.163:0/1106287250 learned_addr learned my addr 172.21.15.163:0/1106287250 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:04.812 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.801+0000 7fb57db5a700 1 -- 172.21.15.163:0/1106287250 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57400b200 con 0x7fb578070430 2023-11-15T04:55:04.812 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.802+0000 7fb57db5a700 1 --2- 172.21.15.163:0/1106287250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb578070430 0x7fb57810aa00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fb57400b190 tx=0x7fb574006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:04.812 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.802+0000 7fb5667fc700 1 -- 172.21.15.163:0/1106287250 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb57401b410 con 0x7fb578070430 2023-11-15T04:55:04.813 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.802+0000 7fb57fdbe700 1 -- 172.21.15.163:0/1106287250 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57810b0f0 con 0x7fb578070430 2023-11-15T04:55:04.813 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.802+0000 7fb5667fc700 1 -- 172.21.15.163:0/1106287250 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb57401b9e0 con 0x7fb578070430 2023-11-15T04:55:04.813 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.802+0000 7fb5667fc700 1 -- 172.21.15.163:0/1106287250 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb57401a880 con 0x7fb578070430 2023-11-15T04:55:04.813 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.802+0000 7fb57fdbe700 1 -- 172.21.15.163:0/1106287250 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb57810b500 con 0x7fb578070430 2023-11-15T04:55:04.813 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.802+0000 7fb5667fc700 1 -- 172.21.15.163:0/1106287250 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb57400f070 con 0x7fb578070430 2023-11-15T04:55:04.814 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.803+0000 7fb5667fc700 1 --2- 172.21.15.163:0/1106287250 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fb56002fd50 0x7fb560032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:04.814 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.803+0000 7fb5667fc700 1 -- 172.21.15.163:0/1106287250 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb5740412a0 con 0x7fb578070430 2023-11-15T04:55:04.814 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.803+0000 7fb57d359700 1 --2- 172.21.15.163:0/1106287250 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fb56002fd50 0x7fb560032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:04.814 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.803+0000 7fb57d359700 1 --2- 172.21.15.163:0/1106287250 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fb56002fd50 0x7fb560032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fb5680096a0 tx=0x7fb568006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:04.815 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.804+0000 7fb57fdbe700 1 -- 172.21.15.163:0/1106287250 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb578059100 con 0x7fb578070430 2023-11-15T04:55:04.815 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.808+0000 7fb5667fc700 1 -- 172.21.15.163:0/1106287250 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb574016070 con 0x7fb578070430 2023-11-15T04:55:04.923 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:04 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:03.706733+0000 mgr.smithi163.qesyrx (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:55:04.923 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:04 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cephadm 2023-11-15T04:55:03.707262+0000 mgr.smithi163.qesyrx (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2023-11-15T04:55:04.923 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:04 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:03.709353+0000 mon.smithi163 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' 2023-11-15T04:55:04.997 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:04.997+0000 7fb57fdbe700 1 -- 172.21.15.163:0/1106287250 --> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5781048f0 con 0x7fb56002fd50 2023-11-15T04:55:05.001 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.001+0000 7fb5667fc700 1 -- 172.21.15.163:0/1106287250 <== mgr.14114 v2:172.21.15.163:6800/1606811720 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fb5781048f0 con 0x7fb56002fd50 2023-11-15T04:55:05.002 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: Scheduled crash update... 2023-11-15T04:55:05.004 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.003+0000 7fb57fdbe700 1 -- 172.21.15.163:0/1106287250 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fb56002fd50 msgr2=0x7fb560032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:05.004 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.003+0000 7fb57fdbe700 1 --2- 172.21.15.163:0/1106287250 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fb56002fd50 0x7fb560032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fb5680096a0 tx=0x7fb568006b40).stop 2023-11-15T04:55:05.005 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.003+0000 7fb57fdbe700 1 -- 172.21.15.163:0/1106287250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb578070430 msgr2=0x7fb57810aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:05.005 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.003+0000 7fb57fdbe700 1 --2- 172.21.15.163:0/1106287250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb578070430 0x7fb57810aa00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fb57400b190 tx=0x7fb574006ee0).stop 2023-11-15T04:55:05.005 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.003+0000 7fb57fdbe700 1 -- 172.21.15.163:0/1106287250 shutdown_connections 2023-11-15T04:55:05.005 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.003+0000 7fb57fdbe700 1 --2- 172.21.15.163:0/1106287250 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7fb56002fd50 0x7fb560032210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:05.006 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.003+0000 7fb57fdbe700 1 --2- 172.21.15.163:0/1106287250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb578070430 0x7fb57810aa00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:05.006 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.003+0000 7fb57fdbe700 1 -- 172.21.15.163:0/1106287250 >> 172.21.15.163:0/1106287250 conn(0x7fb5780f63c0 msgr2=0x7fb5780f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:05.006 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.003+0000 7fb57fdbe700 1 -- 172.21.15.163:0/1106287250 shutdown_connections 2023-11-15T04:55:05.006 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.003+0000 7fb57fdbe700 1 -- 172.21.15.163:0/1106287250 wait complete. 2023-11-15T04:55:05.483 INFO:teuthology.orchestra.run.smithi163.stderr:Enabling mgr prometheus module... 2023-11-15T04:55:05.484 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2023-11-15T04:55:05.859 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.858+0000 7f998c1c0700 1 Processor -- start 2023-11-15T04:55:05.860 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.858+0000 7f998c1c0700 1 -- start start 2023-11-15T04:55:05.860 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.858+0000 7f998c1c0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9984070430 0x7f99840fc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:05.860 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.858+0000 7f998c1c0700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99840fcbc0 con 0x7f9984070430 2023-11-15T04:55:05.860 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.858+0000 7f9989f5c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9984070430 0x7f99840fc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:05.860 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.858+0000 7f9989f5c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9984070430 0x7f99840fc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36916/0 (socket says 172.21.15.163:36916) 2023-11-15T04:55:05.861 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.858+0000 7f9989f5c700 1 -- 172.21.15.163:0/3307615054 learned_addr learned my addr 172.21.15.163:0/3307615054 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:05.863 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.858+0000 7f9989f5c700 1 -- 172.21.15.163:0/3307615054 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99840fcd00 con 0x7f9984070430 2023-11-15T04:55:05.863 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.859+0000 7f9989f5c700 1 --2- 172.21.15.163:0/3307615054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9984070430 0x7f99840fc6d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f998000b5f0 tx=0x7f9980006ee0).ready entity=mon.0 client_cookie=bf1520790bbc60d1 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:05.863 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.859+0000 7f9988f5a700 1 -- 172.21.15.163:0/3307615054 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f998000d600 con 0x7f9984070430 2023-11-15T04:55:05.863 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.859+0000 7f9988f5a700 1 -- 172.21.15.163:0/3307615054 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f998000dc10 con 0x7f9984070430 2023-11-15T04:55:05.864 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.859+0000 7f9988f5a700 1 -- 172.21.15.163:0/3307615054 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9980011cd0 con 0x7f9984070430 2023-11-15T04:55:05.864 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.859+0000 7f998c1c0700 1 -- 172.21.15.163:0/3307615054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9984070430 msgr2=0x7f99840fc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:05.864 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.859+0000 7f998c1c0700 1 --2- 172.21.15.163:0/3307615054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9984070430 0x7f99840fc6d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f998000b5f0 tx=0x7f9980006ee0).stop 2023-11-15T04:55:05.864 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.859+0000 7f998c1c0700 1 -- 172.21.15.163:0/3307615054 shutdown_connections 2023-11-15T04:55:05.864 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.859+0000 7f998c1c0700 1 --2- 172.21.15.163:0/3307615054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9984070430 0x7f99840fc6d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:05.865 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.859+0000 7f998c1c0700 1 -- 172.21.15.163:0/3307615054 >> 172.21.15.163:0/3307615054 conn(0x7f99840f63c0 msgr2=0x7f99840f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:05.865 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.861+0000 7f998c1c0700 1 -- 172.21.15.163:0/3307615054 shutdown_connections 2023-11-15T04:55:05.865 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.861+0000 7f998c1c0700 1 -- 172.21.15.163:0/3307615054 wait complete. 2023-11-15T04:55:05.865 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.861+0000 7f998c1c0700 1 Processor -- start 2023-11-15T04:55:05.865 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.861+0000 7f998c1c0700 1 -- start start 2023-11-15T04:55:05.866 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.862+0000 7f998c1c0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9984070430 0x7f9984111080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:05.866 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.862+0000 7f998c1c0700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9984111570 con 0x7f9984070430 2023-11-15T04:55:05.866 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.862+0000 7f9989f5c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9984070430 0x7f9984111080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:05.867 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.862+0000 7f9989f5c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9984070430 0x7f9984111080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36932/0 (socket says 172.21.15.163:36932) 2023-11-15T04:55:05.867 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.862+0000 7f9989f5c700 1 -- 172.21.15.163:0/2284067006 learned_addr learned my addr 172.21.15.163:0/2284067006 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:05.867 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.862+0000 7f9989f5c700 1 -- 172.21.15.163:0/2284067006 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998000b200 con 0x7f9984070430 2023-11-15T04:55:05.868 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.862+0000 7f9989f5c700 1 --2- 172.21.15.163:0/2284067006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9984070430 0x7f9984111080 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f998000dfb0 tx=0x7f9980006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:05.868 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.862+0000 7f99727fc700 1 -- 172.21.15.163:0/2284067006 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f998001b410 con 0x7f9984070430 2023-11-15T04:55:05.868 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.862+0000 7f99727fc700 1 -- 172.21.15.163:0/2284067006 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f998001ba20 con 0x7f9984070430 2023-11-15T04:55:05.868 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.862+0000 7f99727fc700 1 -- 172.21.15.163:0/2284067006 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f998001a970 con 0x7f9984070430 2023-11-15T04:55:05.868 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.862+0000 7f998c1c0700 1 -- 172.21.15.163:0/2284067006 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9984111770 con 0x7f9984070430 2023-11-15T04:55:05.869 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.862+0000 7f998c1c0700 1 -- 172.21.15.163:0/2284067006 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9984111b80 con 0x7f9984070430 2023-11-15T04:55:05.869 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.864+0000 7f998c1c0700 1 -- 172.21.15.163:0/2284067006 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9984045ad0 con 0x7f9984070430 2023-11-15T04:55:05.869 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.864+0000 7f99727fc700 1 -- 172.21.15.163:0/2284067006 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f998000f070 con 0x7f9984070430 2023-11-15T04:55:05.869 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.864+0000 7f99727fc700 1 --2- 172.21.15.163:0/2284067006 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f996c02fdb0 0x7f996c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:05.869 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.864+0000 7f99727fc700 1 -- 172.21.15.163:0/2284067006 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f99800415e0 con 0x7f9984070430 2023-11-15T04:55:05.870 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.864+0000 7f998975b700 1 --2- 172.21.15.163:0/2284067006 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f996c02fdb0 0x7f996c032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:05.870 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.865+0000 7f998975b700 1 --2- 172.21.15.163:0/2284067006 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f996c02fdb0 0x7f996c032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f99740096a0 tx=0x7f9974006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:05.870 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:05.868+0000 7f99727fc700 1 -- 172.21.15.163:0/2284067006 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9980010360 con 0x7f9984070430 2023-11-15T04:55:06.069 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:06.068+0000 7f998c1c0700 1 -- 172.21.15.163:0/2284067006 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f99840fdab0 con 0x7f9984070430 2023-11-15T04:55:06.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:06 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:04.998477+0000 mgr.smithi163.qesyrx (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:55:06.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:06 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cephadm 2023-11-15T04:55:04.999102+0000 mgr.smithi163.qesyrx (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2023-11-15T04:55:06.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:06 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:05.001807+0000 mon.smithi163 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' 2023-11-15T04:55:06.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:06 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:05.275135+0000 mon.smithi163 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' 2023-11-15T04:55:06.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:06 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:05.498469+0000 mon.smithi163 (mon.0) 59 : audit [INF] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' 2023-11-15T04:55:07.006 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.004+0000 7f99727fc700 1 -- 172.21.15.163:0/2284067006 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f998001aad0 con 0x7f9984070430 2023-11-15T04:55:07.007 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.005+0000 7f99727fc700 1 -- 172.21.15.163:0/2284067006 <== mon.0 v2:172.21.15.163:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v8) v1 ==== 89+0+0 (secure 0 0 0) 0x7f9980019af0 con 0x7f9984070430 2023-11-15T04:55:07.009 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.008+0000 7f998c1c0700 1 -- 172.21.15.163:0/2284067006 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f996c02fdb0 msgr2=0x7f996c032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:07.010 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.008+0000 7f998c1c0700 1 --2- 172.21.15.163:0/2284067006 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f996c02fdb0 0x7f996c032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f99740096a0 tx=0x7f9974006b40).stop 2023-11-15T04:55:07.010 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.008+0000 7f998c1c0700 1 -- 172.21.15.163:0/2284067006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9984070430 msgr2=0x7f9984111080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:07.010 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.008+0000 7f998c1c0700 1 --2- 172.21.15.163:0/2284067006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9984070430 0x7f9984111080 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f998000dfb0 tx=0x7f9980006ee0).stop 2023-11-15T04:55:07.010 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.008+0000 7f998c1c0700 1 -- 172.21.15.163:0/2284067006 shutdown_connections 2023-11-15T04:55:07.011 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.008+0000 7f998c1c0700 1 --2- 172.21.15.163:0/2284067006 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f996c02fdb0 0x7f996c032270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:07.011 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.008+0000 7f998c1c0700 1 --2- 172.21.15.163:0/2284067006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9984070430 0x7f9984111080 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:07.012 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.008+0000 7f998c1c0700 1 -- 172.21.15.163:0/2284067006 >> 172.21.15.163:0/2284067006 conn(0x7f99840f63c0 msgr2=0x7f99840f7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:07.012 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.008+0000 7f998c1c0700 1 -- 172.21.15.163:0/2284067006 shutdown_connections 2023-11-15T04:55:07.012 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.008+0000 7f998c1c0700 1 -- 172.21.15.163:0/2284067006 wait complete. 2023-11-15T04:55:07.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:07 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:06.069872+0000 mon.smithi163 (mon.0) 60 : audit [INF] from='client.? 172.21.15.163:0/2284067006' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-11-15T04:55:07.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:07 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:06.319593+0000 mon.smithi163 (mon.0) 61 : audit [INF] from='mgr.14114 172.21.15.163:0/3128270961' entity='mgr.smithi163.qesyrx' 2023-11-15T04:55:07.374 INFO:teuthology.orchestra.run.smithi163.stderr:Deploying prometheus service with default placement... 2023-11-15T04:55:07.374 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2023-11-15T04:55:07.742 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.738+0000 7f90e1779700 1 Processor -- start 2023-11-15T04:55:07.742 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.738+0000 7f90e1779700 1 -- start start 2023-11-15T04:55:07.742 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.739+0000 7f90e1779700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90dc0703f0 0x7f90dc0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:07.742 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.739+0000 7f90e1779700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc0fee10 con 0x7f90dc0703f0 2023-11-15T04:55:07.743 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.739+0000 7f90daffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90dc0703f0 0x7f90dc0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:07.743 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.739+0000 7f90daffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90dc0703f0 0x7f90dc0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36956/0 (socket says 172.21.15.163:36956) 2023-11-15T04:55:07.743 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.739+0000 7f90daffd700 1 -- 172.21.15.163:0/361408205 learned_addr learned my addr 172.21.15.163:0/361408205 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:07.743 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.739+0000 7f90daffd700 1 -- 172.21.15.163:0/361408205 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc0fef50 con 0x7f90dc0703f0 2023-11-15T04:55:07.743 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.739+0000 7f90daffd700 1 --2- 172.21.15.163:0/361408205 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90dc0703f0 0x7f90dc0fe920 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f90cc00b5f0 tx=0x7f90cc006ee0).ready entity=mon.0 client_cookie=b158563277a78533 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:07.744 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.739+0000 7f90d9ffb700 1 -- 172.21.15.163:0/361408205 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90cc00d600 con 0x7f90dc0703f0 2023-11-15T04:55:07.744 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.739+0000 7f90d9ffb700 1 -- 172.21.15.163:0/361408205 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f90cc00dc10 con 0x7f90dc0703f0 2023-11-15T04:55:07.744 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.739+0000 7f90d9ffb700 1 -- 172.21.15.163:0/361408205 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90cc011cd0 con 0x7f90dc0703f0 2023-11-15T04:55:07.744 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.740+0000 7f90e1779700 1 -- 172.21.15.163:0/361408205 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90dc0703f0 msgr2=0x7f90dc0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:07.744 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.740+0000 7f90e1779700 1 --2- 172.21.15.163:0/361408205 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90dc0703f0 0x7f90dc0fe920 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f90cc00b5f0 tx=0x7f90cc006ee0).stop 2023-11-15T04:55:07.745 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.740+0000 7f90e1779700 1 -- 172.21.15.163:0/361408205 shutdown_connections 2023-11-15T04:55:07.745 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.740+0000 7f90e1779700 1 --2- 172.21.15.163:0/361408205 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90dc0703f0 0x7f90dc0fe920 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:07.745 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.740+0000 7f90e1779700 1 -- 172.21.15.163:0/361408205 >> 172.21.15.163:0/361408205 conn(0x7f90dc0f63c0 msgr2=0x7f90dc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:07.745 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.741+0000 7f90e1779700 1 -- 172.21.15.163:0/361408205 shutdown_connections 2023-11-15T04:55:07.745 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.741+0000 7f90e1779700 1 -- 172.21.15.163:0/361408205 wait complete. 2023-11-15T04:55:07.747 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.741+0000 7f90e1779700 1 Processor -- start 2023-11-15T04:55:07.748 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.741+0000 7f90e1779700 1 -- start start 2023-11-15T04:55:07.748 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.741+0000 7f90e1779700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90dc0703f0 0x7f90dc0fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:07.748 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.741+0000 7f90e1779700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc0fa820 con 0x7f90dc0703f0 2023-11-15T04:55:07.749 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.742+0000 7f90daffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90dc0703f0 0x7f90dc0fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:07.749 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.742+0000 7f90daffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90dc0703f0 0x7f90dc0fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36960/0 (socket says 172.21.15.163:36960) 2023-11-15T04:55:07.749 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.742+0000 7f90daffd700 1 -- 172.21.15.163:0/3778148519 learned_addr learned my addr 172.21.15.163:0/3778148519 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:07.749 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.742+0000 7f90daffd700 1 -- 172.21.15.163:0/3778148519 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90cc00b200 con 0x7f90dc0703f0 2023-11-15T04:55:07.749 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.742+0000 7f90daffd700 1 --2- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90dc0703f0 0x7f90dc0fc3a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f90cc006a40 tx=0x7f90cc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:07.750 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.742+0000 7f90c37fe700 1 -- 172.21.15.163:0/3778148519 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90cc011c90 con 0x7f90dc0703f0 2023-11-15T04:55:07.750 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.742+0000 7f90c37fe700 1 -- 172.21.15.163:0/3778148519 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f90cc01b8c0 con 0x7f90dc0703f0 2023-11-15T04:55:07.750 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.742+0000 7f90e1779700 1 -- 172.21.15.163:0/3778148519 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90dc0fa9c0 con 0x7f90dc0703f0 2023-11-15T04:55:07.750 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.742+0000 7f90c37fe700 1 -- 172.21.15.163:0/3778148519 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90cc01a8c0 con 0x7f90dc0703f0 2023-11-15T04:55:07.751 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.742+0000 7f90e1779700 1 -- 172.21.15.163:0/3778148519 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90dc0fadd0 con 0x7f90dc0703f0 2023-11-15T04:55:07.751 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.743+0000 7f90c37fe700 1 -- 172.21.15.163:0/3778148519 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f90cc00f070 con 0x7f90dc0703f0 2023-11-15T04:55:07.751 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.743+0000 7f90e1779700 1 -- 172.21.15.163:0/3778148519 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90dc045ad0 con 0x7f90dc0703f0 2023-11-15T04:55:07.751 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.744+0000 7f90c37fe700 1 --2- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f90c802fe00 0x7f90c80322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:07.751 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.744+0000 7f90c37fe700 1 -- 172.21.15.163:0/3778148519 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f90cc040810 con 0x7f90dc0703f0 2023-11-15T04:55:07.752 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.744+0000 7f90da7fc700 1 -- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f90c802fe00 msgr2=0x7f90c80322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.163:6800/1606811720 2023-11-15T04:55:07.752 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.744+0000 7f90da7fc700 1 --2- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f90c802fe00 0x7f90c80322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T04:55:07.752 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.748+0000 7f90c37fe700 1 -- 172.21.15.163:0/3778148519 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f90cc019b80 con 0x7f90dc0703f0 2023-11-15T04:55:07.934 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.932+0000 7f90e1779700 1 -- 172.21.15.163:0/3778148519 --> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f90dc0ffdf0 con 0x7f90c802fe00 2023-11-15T04:55:07.946 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.945+0000 7f90da7fc700 1 -- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f90c802fe00 msgr2=0x7f90c80322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.163:6800/1606811720 2023-11-15T04:55:07.946 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:07.945+0000 7f90da7fc700 1 --2- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f90c802fe00 0x7f90c80322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T04:55:08.347 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:08.346+0000 7f90da7fc700 1 -- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f90c802fe00 msgr2=0x7f90c80322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.163:6800/1606811720 2023-11-15T04:55:08.347 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:08.346+0000 7f90da7fc700 1 --2- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f90c802fe00 0x7f90c80322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T04:55:08.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:08 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:07.007091+0000 mon.smithi163 (mon.0) 62 : audit [INF] from='client.? 172.21.15.163:0/2284067006' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-11-15T04:55:08.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:08 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:07.007139+0000 mon.smithi163 (mon.0) 63 : cluster [DBG] mgrmap e8: smithi163.qesyrx(active, since 16s) 2023-11-15T04:55:09.148 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:09.147+0000 7f90da7fc700 1 -- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f90c802fe00 msgr2=0x7f90c80322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.163:6800/1606811720 2023-11-15T04:55:09.148 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:09.147+0000 7f90da7fc700 1 --2- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f90c802fe00 0x7f90c80322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T04:55:10.750 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:10.748+0000 7f90da7fc700 1 -- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f90c802fe00 msgr2=0x7f90c80322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.163:6800/1606811720 2023-11-15T04:55:10.750 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:10.748+0000 7f90da7fc700 1 --2- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f90c802fe00 0x7f90c80322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-15T04:55:12.727 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:12.725+0000 7f90c37fe700 1 -- 172.21.15.163:0/3778148519 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mgrmap(e 9) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f90cc01ba30 con 0x7f90dc0703f0 2023-11-15T04:55:12.728 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:12.725+0000 7f90c37fe700 1 -- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f90c802fe00 msgr2=0x7f90c80322c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T04:55:12.728 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:12.725+0000 7f90c37fe700 1 --2- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f90c802fe00 0x7f90c80322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:55:13.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:12 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:11.952143+0000 mon.smithi163 (mon.0) 64 : cluster [INF] Active manager daemon smithi163.qesyrx restarted 2023-11-15T04:55:13.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:12 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:11.952606+0000 mon.smithi163 (mon.0) 65 : cluster [INF] Activating manager daemon smithi163.qesyrx 2023-11-15T04:55:13.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:12 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:11.954280+0000 mon.smithi163 (mon.0) 66 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-11-15T04:55:13.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:12 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:12.728312+0000 mon.smithi163 (mon.0) 67 : cluster [DBG] mgrmap e9: smithi163.qesyrx(active, starting, since 0.775812s) 2023-11-15T04:55:13.204 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:12 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:12.729499+0000 mon.smithi163 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.163:0/2416123149' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2023-11-15T04:55:13.204 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:12 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:12.730163+0000 mon.smithi163 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.163:0/2416123149' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "mgr metadata", "who": "smithi163.qesyrx", "id": "smithi163.qesyrx"}]: dispatch 2023-11-15T04:55:13.204 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:12 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:12.731255+0000 mon.smithi163 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.163:0/2416123149' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T04:55:13.204 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:12 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:12.731498+0000 mon.smithi163 (mon.0) 71 : audit [DBG] from='mgr.14142 172.21.15.163:0/2416123149' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T04:55:13.204 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:12 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:12.731707+0000 mon.smithi163 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.163:0/2416123149' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T04:55:13.205 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:12 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:12.745876+0000 mon.smithi163 (mon.0) 73 : cluster [INF] Manager daemon smithi163.qesyrx is now available 2023-11-15T04:55:13.205 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:12 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:12.850408+0000 mon.smithi163 (mon.0) 74 : audit [DBG] from='mgr.14142 172.21.15.163:0/2416123149' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:55:13.205 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:12 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:12.851775+0000 mon.smithi163 (mon.0) 75 : audit [DBG] from='mgr.14142 172.21.15.163:0/2416123149' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:55:13.205 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:12 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:12.866544+0000 mon.smithi163 (mon.0) 76 : audit [INF] from='mgr.14142 172.21.15.163:0/2416123149' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:55:13.205 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:12 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:12.869025+0000 mon.smithi163 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.163:0/2416123149' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/trash_purge_schedule"}]: dispatch 2023-11-15T04:55:13.728 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:13.726+0000 7f90c37fe700 1 -- 172.21.15.163:0/3778148519 <== mon.0 v2:172.21.15.163:3300/0 8 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f90cc03d310 con 0x7f90dc0703f0 2023-11-15T04:55:13.728 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:13.726+0000 7f90c37fe700 1 --2- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f90c8033c20 0x7f90c8036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:13.729 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:13.726+0000 7f90c37fe700 1 -- 172.21.15.163:0/3778148519 --> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f90c8036500 con 0x7f90c8033c20 2023-11-15T04:55:13.729 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:13.727+0000 7f90da7fc700 1 --2- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f90c8033c20 0x7f90c8036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:13.729 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:13.727+0000 7f90da7fc700 1 --2- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f90c8033c20 0x7f90c8036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f90d0006fd0 tx=0x7f90d0006d30).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:13.736 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:13.734+0000 7f90c37fe700 1 -- 172.21.15.163:0/3778148519 <== mgr.14142 v2:172.21.15.163:6800/2933442915 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f90c8036500 con 0x7f90c8033c20 2023-11-15T04:55:13.736 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-11-15T04:55:13.739 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:13.736+0000 7f90e1779700 1 -- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f90c8033c20 msgr2=0x7f90c8036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:13.739 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:13.736+0000 7f90e1779700 1 --2- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f90c8033c20 0x7f90c8036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f90d0006fd0 tx=0x7f90d0006d30).stop 2023-11-15T04:55:13.739 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:13.736+0000 7f90e1779700 1 -- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90dc0703f0 msgr2=0x7f90dc0fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:13.739 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:13.736+0000 7f90e1779700 1 --2- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90dc0703f0 0x7f90dc0fc3a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f90cc006a40 tx=0x7f90cc006ee0).stop 2023-11-15T04:55:13.739 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:13.736+0000 7f90e1779700 1 -- 172.21.15.163:0/3778148519 shutdown_connections 2023-11-15T04:55:13.740 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:13.736+0000 7f90e1779700 1 --2- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f90c8033c20 0x7f90c8036010 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:13.740 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:13.736+0000 7f90e1779700 1 --2- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f90dc0703f0 0x7f90dc0fc3a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:13.741 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:13.736+0000 7f90e1779700 1 --2- 172.21.15.163:0/3778148519 >> [v2:172.21.15.163:6800/1606811720,v1:172.21.15.163:6801/1606811720] conn(0x7f90c802fe00 0x7f90c80322c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:55:13.741 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:13.736+0000 7f90e1779700 1 -- 172.21.15.163:0/3778148519 >> 172.21.15.163:0/3778148519 conn(0x7f90dc0f63c0 msgr2=0x7f90dc0f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:13.741 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:13.737+0000 7f90e1779700 1 -- 172.21.15.163:0/3778148519 shutdown_connections 2023-11-15T04:55:13.741 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:13.737+0000 7f90e1779700 1 -- 172.21.15.163:0/3778148519 wait complete. 2023-11-15T04:55:14.132 INFO:teuthology.orchestra.run.smithi163.stderr:Deploying grafana service with default placement... 2023-11-15T04:55:14.133 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2023-11-15T04:55:14.563 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.558+0000 7f259698e700 1 Processor -- start 2023-11-15T04:55:14.563 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.558+0000 7f259698e700 1 -- start start 2023-11-15T04:55:14.563 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.559+0000 7f259698e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25900fb1e0 0x7f25900fb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:14.564 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.559+0000 7f259698e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25900fbaf0 con 0x7f25900fb1e0 2023-11-15T04:55:14.564 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.559+0000 7f258ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25900fb1e0 0x7f25900fb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:14.564 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.559+0000 7f258ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25900fb1e0 0x7f25900fb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:46882/0 (socket says 172.21.15.163:46882) 2023-11-15T04:55:14.564 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.559+0000 7f258ffff700 1 -- 172.21.15.163:0/2298466028 learned_addr learned my addr 172.21.15.163:0/2298466028 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:14.564 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.559+0000 7f258ffff700 1 -- 172.21.15.163:0/2298466028 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25900fbc30 con 0x7f25900fb1e0 2023-11-15T04:55:14.565 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.559+0000 7f258ffff700 1 --2- 172.21.15.163:0/2298466028 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25900fb1e0 0x7f25900fb600 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f258000b5f0 tx=0x7f2580006ee0).ready entity=mon.0 client_cookie=fe1f054754aef3f8 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:14.565 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.560+0000 7f258effd700 1 -- 172.21.15.163:0/2298466028 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f258000d600 con 0x7f25900fb1e0 2023-11-15T04:55:14.565 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.560+0000 7f258effd700 1 -- 172.21.15.163:0/2298466028 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f258000dc10 con 0x7f25900fb1e0 2023-11-15T04:55:14.565 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.560+0000 7f258effd700 1 -- 172.21.15.163:0/2298466028 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2580011cd0 con 0x7f25900fb1e0 2023-11-15T04:55:14.565 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.560+0000 7f259698e700 1 -- 172.21.15.163:0/2298466028 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25900fb1e0 msgr2=0x7f25900fb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:14.566 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.560+0000 7f259698e700 1 --2- 172.21.15.163:0/2298466028 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25900fb1e0 0x7f25900fb600 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f258000b5f0 tx=0x7f2580006ee0).stop 2023-11-15T04:55:14.566 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.560+0000 7f259698e700 1 -- 172.21.15.163:0/2298466028 shutdown_connections 2023-11-15T04:55:14.566 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.560+0000 7f259698e700 1 --2- 172.21.15.163:0/2298466028 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25900fb1e0 0x7f25900fb600 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:14.566 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.560+0000 7f259698e700 1 -- 172.21.15.163:0/2298466028 >> 172.21.15.163:0/2298466028 conn(0x7f25900f5b10 msgr2=0x7f25900f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:14.566 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.560+0000 7f259698e700 1 -- 172.21.15.163:0/2298466028 shutdown_connections 2023-11-15T04:55:14.567 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.560+0000 7f259698e700 1 -- 172.21.15.163:0/2298466028 wait complete. 2023-11-15T04:55:14.567 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.561+0000 7f259698e700 1 Processor -- start 2023-11-15T04:55:14.567 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.561+0000 7f259698e700 1 -- start start 2023-11-15T04:55:14.567 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.561+0000 7f259698e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25900fb1e0 0x7f259010aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:14.567 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.561+0000 7f259698e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259010af50 con 0x7f25900fb1e0 2023-11-15T04:55:14.568 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.561+0000 7f258ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25900fb1e0 0x7f259010aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:14.568 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.561+0000 7f258ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25900fb1e0 0x7f259010aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:46898/0 (socket says 172.21.15.163:46898) 2023-11-15T04:55:14.568 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.561+0000 7f258ffff700 1 -- 172.21.15.163:0/3143110937 learned_addr learned my addr 172.21.15.163:0/3143110937 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:14.568 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.562+0000 7f258ffff700 1 -- 172.21.15.163:0/3143110937 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258000b200 con 0x7f25900fb1e0 2023-11-15T04:55:14.568 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.562+0000 7f258ffff700 1 --2- 172.21.15.163:0/3143110937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25900fb1e0 0x7f259010aa60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f258000dfb0 tx=0x7f2580006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:14.569 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.562+0000 7f258cff9700 1 -- 172.21.15.163:0/3143110937 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2580011df0 con 0x7f25900fb1e0 2023-11-15T04:55:14.570 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.562+0000 7f258cff9700 1 -- 172.21.15.163:0/3143110937 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f258001b8c0 con 0x7f25900fb1e0 2023-11-15T04:55:14.570 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.562+0000 7f258cff9700 1 -- 172.21.15.163:0/3143110937 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f258001a8c0 con 0x7f25900fb1e0 2023-11-15T04:55:14.570 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.562+0000 7f259698e700 1 -- 172.21.15.163:0/3143110937 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f259010b150 con 0x7f25900fb1e0 2023-11-15T04:55:14.570 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.562+0000 7f259698e700 1 -- 172.21.15.163:0/3143110937 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f259010b560 con 0x7f25900fb1e0 2023-11-15T04:55:14.571 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.563+0000 7f258cff9700 1 -- 172.21.15.163:0/3143110937 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f258000f070 con 0x7f25900fb1e0 2023-11-15T04:55:14.571 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.563+0000 7f258cff9700 1 --2- 172.21.15.163:0/3143110937 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f257002fcf0 0x7f25700321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:14.571 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.563+0000 7f258cff9700 1 -- 172.21.15.163:0/3143110937 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f2580040fe0 con 0x7f25900fb1e0 2023-11-15T04:55:14.571 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.563+0000 7f259698e700 1 -- 172.21.15.163:0/3143110937 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2590045ad0 con 0x7f25900fb1e0 2023-11-15T04:55:14.571 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.563+0000 7f258f7fe700 1 --2- 172.21.15.163:0/3143110937 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f257002fcf0 0x7f25700321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:14.572 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.563+0000 7f258f7fe700 1 --2- 172.21.15.163:0/3143110937 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f257002fcf0 0x7f25700321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f25780096a0 tx=0x7f2578006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:14.572 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.567+0000 7f258cff9700 1 -- 172.21.15.163:0/3143110937 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2580016070 con 0x7f25900fb1e0 2023-11-15T04:55:14.736 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.734+0000 7f258cff9700 1 -- 172.21.15.163:0/3143110937 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f2580024790 con 0x7f25900fb1e0 2023-11-15T04:55:14.751 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.749+0000 7f259698e700 1 -- 172.21.15.163:0/3143110937 --> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f259010be40 con 0x7f257002fcf0 2023-11-15T04:55:14.754 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.753+0000 7f258cff9700 1 -- 172.21.15.163:0/3143110937 <== mgr.14142 v2:172.21.15.163:6800/2933442915 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f259010be40 con 0x7f257002fcf0 2023-11-15T04:55:14.754 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-11-15T04:55:14.757 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.755+0000 7f259698e700 1 -- 172.21.15.163:0/3143110937 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f257002fcf0 msgr2=0x7f25700321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:14.757 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.755+0000 7f259698e700 1 --2- 172.21.15.163:0/3143110937 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f257002fcf0 0x7f25700321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f25780096a0 tx=0x7f2578006b40).stop 2023-11-15T04:55:14.757 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.755+0000 7f259698e700 1 -- 172.21.15.163:0/3143110937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25900fb1e0 msgr2=0x7f259010aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:14.757 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.755+0000 7f259698e700 1 --2- 172.21.15.163:0/3143110937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25900fb1e0 0x7f259010aa60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f258000dfb0 tx=0x7f2580006ee0).stop 2023-11-15T04:55:14.758 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.755+0000 7f259698e700 1 -- 172.21.15.163:0/3143110937 shutdown_connections 2023-11-15T04:55:14.758 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.755+0000 7f259698e700 1 --2- 172.21.15.163:0/3143110937 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f257002fcf0 0x7f25700321b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:14.758 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.755+0000 7f259698e700 1 --2- 172.21.15.163:0/3143110937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25900fb1e0 0x7f259010aa60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:14.758 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.755+0000 7f259698e700 1 -- 172.21.15.163:0/3143110937 >> 172.21.15.163:0/3143110937 conn(0x7f25900f5b10 msgr2=0x7f25900fa780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:14.759 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.755+0000 7f259698e700 1 -- 172.21.15.163:0/3143110937 shutdown_connections 2023-11-15T04:55:14.759 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:14.755+0000 7f259698e700 1 -- 172.21.15.163:0/3143110937 wait complete. 2023-11-15T04:55:14.953 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:14 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:13.644170+0000 mon.smithi163 (mon.0) 78 : audit [INF] from='mgr.14142 172.21.15.163:0/2416123149' entity='mgr.smithi163.qesyrx' 2023-11-15T04:55:14.953 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:14 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:13.729855+0000 mon.smithi163 (mon.0) 79 : cluster [DBG] mgrmap e10: smithi163.qesyrx(active, since 1.77736s) 2023-11-15T04:55:14.953 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:14 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cephadm 2023-11-15T04:55:13.730882+0000 mgr.smithi163.qesyrx (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-11-15T04:55:14.953 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:14 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:13.735703+0000 mon.smithi163 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.163:0/2416123149' entity='mgr.smithi163.qesyrx' 2023-11-15T04:55:15.128 INFO:teuthology.orchestra.run.smithi163.stderr:Deploying node-exporter service with default placement... 2023-11-15T04:55:15.128 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2023-11-15T04:55:15.461 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.457+0000 7fc372a9e700 1 Processor -- start 2023-11-15T04:55:15.461 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.457+0000 7fc372a9e700 1 -- start start 2023-11-15T04:55:15.461 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.457+0000 7fc372a9e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc36c070430 0x7fc36c0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:15.462 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.457+0000 7fc372a9e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c0fedd0 con 0x7fc36c070430 2023-11-15T04:55:15.462 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.458+0000 7fc37083a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc36c070430 0x7fc36c0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:15.462 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.458+0000 7fc37083a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc36c070430 0x7fc36c0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:46914/0 (socket says 172.21.15.163:46914) 2023-11-15T04:55:15.462 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.458+0000 7fc37083a700 1 -- 172.21.15.163:0/700763812 learned_addr learned my addr 172.21.15.163:0/700763812 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:15.462 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.458+0000 7fc37083a700 1 -- 172.21.15.163:0/700763812 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c0fef10 con 0x7fc36c070430 2023-11-15T04:55:15.463 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.458+0000 7fc37083a700 1 --2- 172.21.15.163:0/700763812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc36c070430 0x7fc36c0fe8e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fc36000b5f0 tx=0x7fc360006ee0).ready entity=mon.0 client_cookie=f7347c3750c1a83 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:15.463 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.458+0000 7fc36b7fe700 1 -- 172.21.15.163:0/700763812 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc36000d600 con 0x7fc36c070430 2023-11-15T04:55:15.463 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.458+0000 7fc36b7fe700 1 -- 172.21.15.163:0/700763812 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc36000dc10 con 0x7fc36c070430 2023-11-15T04:55:15.463 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.458+0000 7fc36b7fe700 1 -- 172.21.15.163:0/700763812 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc360011cd0 con 0x7fc36c070430 2023-11-15T04:55:15.464 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.459+0000 7fc372a9e700 1 -- 172.21.15.163:0/700763812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc36c070430 msgr2=0x7fc36c0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:15.464 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.459+0000 7fc372a9e700 1 --2- 172.21.15.163:0/700763812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc36c070430 0x7fc36c0fe8e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fc36000b5f0 tx=0x7fc360006ee0).stop 2023-11-15T04:55:15.464 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.459+0000 7fc372a9e700 1 -- 172.21.15.163:0/700763812 shutdown_connections 2023-11-15T04:55:15.464 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.459+0000 7fc372a9e700 1 --2- 172.21.15.163:0/700763812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc36c070430 0x7fc36c0fe8e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:15.464 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.459+0000 7fc372a9e700 1 -- 172.21.15.163:0/700763812 >> 172.21.15.163:0/700763812 conn(0x7fc36c0f63c0 msgr2=0x7fc36c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:15.465 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.459+0000 7fc372a9e700 1 -- 172.21.15.163:0/700763812 shutdown_connections 2023-11-15T04:55:15.465 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.459+0000 7fc372a9e700 1 -- 172.21.15.163:0/700763812 wait complete. 2023-11-15T04:55:15.465 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.460+0000 7fc372a9e700 1 Processor -- start 2023-11-15T04:55:15.465 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.460+0000 7fc372a9e700 1 -- start start 2023-11-15T04:55:15.467 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.460+0000 7fc372a9e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc36c070430 0x7fc36c10ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:15.467 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.460+0000 7fc372a9e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c10aff0 con 0x7fc36c070430 2023-11-15T04:55:15.468 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.460+0000 7fc37083a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc36c070430 0x7fc36c10ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:15.468 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.461+0000 7fc37083a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc36c070430 0x7fc36c10ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:46922/0 (socket says 172.21.15.163:46922) 2023-11-15T04:55:15.468 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.461+0000 7fc37083a700 1 -- 172.21.15.163:0/891737782 learned_addr learned my addr 172.21.15.163:0/891737782 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:15.468 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.461+0000 7fc37083a700 1 -- 172.21.15.163:0/891737782 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36000b200 con 0x7fc36c070430 2023-11-15T04:55:15.468 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.461+0000 7fc37083a700 1 --2- 172.21.15.163:0/891737782 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc36c070430 0x7fc36c10ab00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fc360006800 tx=0x7fc360006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:15.469 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.461+0000 7fc3697fa700 1 -- 172.21.15.163:0/891737782 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc360011df0 con 0x7fc36c070430 2023-11-15T04:55:15.469 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.461+0000 7fc3697fa700 1 -- 172.21.15.163:0/891737782 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc36001b8c0 con 0x7fc36c070430 2023-11-15T04:55:15.469 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.461+0000 7fc372a9e700 1 -- 172.21.15.163:0/891737782 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc36c10b1f0 con 0x7fc36c070430 2023-11-15T04:55:15.469 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.461+0000 7fc3697fa700 1 -- 172.21.15.163:0/891737782 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc36001a830 con 0x7fc36c070430 2023-11-15T04:55:15.469 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.461+0000 7fc372a9e700 1 -- 172.21.15.163:0/891737782 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc36c10b600 con 0x7fc36c070430 2023-11-15T04:55:15.470 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.462+0000 7fc3697fa700 1 -- 172.21.15.163:0/891737782 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fc36000f070 con 0x7fc36c070430 2023-11-15T04:55:15.470 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.462+0000 7fc3697fa700 1 --2- 172.21.15.163:0/891737782 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fc35002feb0 0x7fc350032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:15.470 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.462+0000 7fc3697fa700 1 -- 172.21.15.163:0/891737782 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fc360040f90 con 0x7fc36c070430 2023-11-15T04:55:15.470 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.462+0000 7fc36bfff700 1 --2- 172.21.15.163:0/891737782 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fc35002feb0 0x7fc350032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:15.470 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.462+0000 7fc372a9e700 1 -- 172.21.15.163:0/891737782 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36c059100 con 0x7fc36c070430 2023-11-15T04:55:15.471 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.462+0000 7fc36bfff700 1 --2- 172.21.15.163:0/891737782 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fc35002feb0 0x7fc350032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fc3580096a0 tx=0x7fc358006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:15.471 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.467+0000 7fc3697fa700 1 -- 172.21.15.163:0/891737782 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc360024410 con 0x7fc36c070430 2023-11-15T04:55:15.644 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.643+0000 7fc372a9e700 1 -- 172.21.15.163:0/891737782 --> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fc36c104960 con 0x7fc35002feb0 2023-11-15T04:55:15.648 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.647+0000 7fc3697fa700 1 -- 172.21.15.163:0/891737782 <== mgr.14142 v2:172.21.15.163:6800/2933442915 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fc36c104960 con 0x7fc35002feb0 2023-11-15T04:55:15.648 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-11-15T04:55:15.651 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.648+0000 7fc372a9e700 1 -- 172.21.15.163:0/891737782 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fc35002feb0 msgr2=0x7fc350032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:15.651 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.648+0000 7fc372a9e700 1 --2- 172.21.15.163:0/891737782 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fc35002feb0 0x7fc350032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fc3580096a0 tx=0x7fc358006b40).stop 2023-11-15T04:55:15.651 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.648+0000 7fc372a9e700 1 -- 172.21.15.163:0/891737782 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc36c070430 msgr2=0x7fc36c10ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:15.651 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.648+0000 7fc372a9e700 1 --2- 172.21.15.163:0/891737782 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc36c070430 0x7fc36c10ab00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fc360006800 tx=0x7fc360006ee0).stop 2023-11-15T04:55:15.651 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.649+0000 7fc372a9e700 1 -- 172.21.15.163:0/891737782 shutdown_connections 2023-11-15T04:55:15.652 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.649+0000 7fc372a9e700 1 --2- 172.21.15.163:0/891737782 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fc35002feb0 0x7fc350032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:15.652 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.649+0000 7fc372a9e700 1 --2- 172.21.15.163:0/891737782 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc36c070430 0x7fc36c10ab00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:15.652 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.649+0000 7fc372a9e700 1 -- 172.21.15.163:0/891737782 >> 172.21.15.163:0/891737782 conn(0x7fc36c0f63c0 msgr2=0x7fc36c0f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:15.652 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.649+0000 7fc372a9e700 1 -- 172.21.15.163:0/891737782 shutdown_connections 2023-11-15T04:55:15.652 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:15.649+0000 7fc372a9e700 1 -- 172.21.15.163:0/891737782 wait complete. 2023-11-15T04:55:15.925 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:15 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:14.737411+0000 mon.smithi163 (mon.0) 81 : cluster [DBG] mgrmap e11: smithi163.qesyrx(active, since 2s) 2023-11-15T04:55:15.926 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:15 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:14.751851+0000 mgr.smithi163.qesyrx (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:55:15.926 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:15 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cephadm 2023-11-15T04:55:14.752336+0000 mgr.smithi163.qesyrx (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2023-11-15T04:55:15.926 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:15 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:14.754360+0000 mon.smithi163 (mon.0) 82 : audit [INF] from='mgr.14142 172.21.15.163:0/2416123149' entity='mgr.smithi163.qesyrx' 2023-11-15T04:55:15.926 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:15 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:15.648325+0000 mon.smithi163 (mon.0) 83 : audit [INF] from='mgr.14142 172.21.15.163:0/2416123149' entity='mgr.smithi163.qesyrx' 2023-11-15T04:55:15.981 INFO:teuthology.orchestra.run.smithi163.stderr:Deploying alertmanager service with default placement... 2023-11-15T04:55:15.982 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2023-11-15T04:55:16.335 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.332+0000 7fe7609a6700 1 Processor -- start 2023-11-15T04:55:16.336 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.332+0000 7fe7609a6700 1 -- start start 2023-11-15T04:55:16.336 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.332+0000 7fe7609a6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c0703f0 0x7fe75c0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:16.336 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.332+0000 7fe7609a6700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c0fee10 con 0x7fe75c0703f0 2023-11-15T04:55:16.336 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.332+0000 7fe75a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c0703f0 0x7fe75c0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:16.336 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.332+0000 7fe75a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c0703f0 0x7fe75c0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:46936/0 (socket says 172.21.15.163:46936) 2023-11-15T04:55:16.337 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.333+0000 7fe75a59c700 1 -- 172.21.15.163:0/3555210515 learned_addr learned my addr 172.21.15.163:0/3555210515 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:16.337 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.333+0000 7fe75a59c700 1 -- 172.21.15.163:0/3555210515 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75c0fef50 con 0x7fe75c0703f0 2023-11-15T04:55:16.337 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.333+0000 7fe75a59c700 1 --2- 172.21.15.163:0/3555210515 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c0703f0 0x7fe75c0fe920 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fe75000b5f0 tx=0x7fe750006ee0).ready entity=mon.0 client_cookie=44342750e732db53 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:16.337 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.333+0000 7fe75959a700 1 -- 172.21.15.163:0/3555210515 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe75000d600 con 0x7fe75c0703f0 2023-11-15T04:55:16.337 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.333+0000 7fe75959a700 1 -- 172.21.15.163:0/3555210515 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe75000dc10 con 0x7fe75c0703f0 2023-11-15T04:55:16.338 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.333+0000 7fe75959a700 1 -- 172.21.15.163:0/3555210515 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe750011cd0 con 0x7fe75c0703f0 2023-11-15T04:55:16.338 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.334+0000 7fe7609a6700 1 -- 172.21.15.163:0/3555210515 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c0703f0 msgr2=0x7fe75c0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:16.338 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.334+0000 7fe7609a6700 1 --2- 172.21.15.163:0/3555210515 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c0703f0 0x7fe75c0fe920 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fe75000b5f0 tx=0x7fe750006ee0).stop 2023-11-15T04:55:16.338 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.334+0000 7fe7609a6700 1 -- 172.21.15.163:0/3555210515 shutdown_connections 2023-11-15T04:55:16.338 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.334+0000 7fe7609a6700 1 --2- 172.21.15.163:0/3555210515 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c0703f0 0x7fe75c0fe920 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:16.339 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.334+0000 7fe7609a6700 1 -- 172.21.15.163:0/3555210515 >> 172.21.15.163:0/3555210515 conn(0x7fe75c0f63c0 msgr2=0x7fe75c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:16.339 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.334+0000 7fe7609a6700 1 -- 172.21.15.163:0/3555210515 shutdown_connections 2023-11-15T04:55:16.339 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.334+0000 7fe7609a6700 1 -- 172.21.15.163:0/3555210515 wait complete. 2023-11-15T04:55:16.339 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.334+0000 7fe7609a6700 1 Processor -- start 2023-11-15T04:55:16.341 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.334+0000 7fe7609a6700 1 -- start start 2023-11-15T04:55:16.342 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.335+0000 7fe7609a6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c0703f0 0x7fe75c10aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:16.342 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.335+0000 7fe7609a6700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c10afc0 con 0x7fe75c0703f0 2023-11-15T04:55:16.342 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.335+0000 7fe75a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c0703f0 0x7fe75c10aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:16.342 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.335+0000 7fe75a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c0703f0 0x7fe75c10aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:46952/0 (socket says 172.21.15.163:46952) 2023-11-15T04:55:16.342 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.335+0000 7fe75a59c700 1 -- 172.21.15.163:0/1364347304 learned_addr learned my addr 172.21.15.163:0/1364347304 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:16.343 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.335+0000 7fe75a59c700 1 -- 172.21.15.163:0/1364347304 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75000b200 con 0x7fe75c0703f0 2023-11-15T04:55:16.343 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.335+0000 7fe75a59c700 1 --2- 172.21.15.163:0/1364347304 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c0703f0 0x7fe75c10aad0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fe75000df90 tx=0x7fe750006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:16.343 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.335+0000 7fe74affd700 1 -- 172.21.15.163:0/1364347304 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe750011da0 con 0x7fe75c0703f0 2023-11-15T04:55:16.343 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.335+0000 7fe7609a6700 1 -- 172.21.15.163:0/1364347304 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe75c10b1c0 con 0x7fe75c0703f0 2023-11-15T04:55:16.343 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.335+0000 7fe74affd700 1 -- 172.21.15.163:0/1364347304 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe75001b8c0 con 0x7fe75c0703f0 2023-11-15T04:55:16.344 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.335+0000 7fe74affd700 1 -- 172.21.15.163:0/1364347304 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe75001a930 con 0x7fe75c0703f0 2023-11-15T04:55:16.344 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.335+0000 7fe7609a6700 1 -- 172.21.15.163:0/1364347304 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75c10b5d0 con 0x7fe75c0703f0 2023-11-15T04:55:16.344 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.336+0000 7fe7609a6700 1 -- 172.21.15.163:0/1364347304 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe75c045ad0 con 0x7fe75c0703f0 2023-11-15T04:55:16.344 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.336+0000 7fe74affd700 1 -- 172.21.15.163:0/1364347304 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fe75000f070 con 0x7fe75c0703f0 2023-11-15T04:55:16.344 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.336+0000 7fe74affd700 1 --2- 172.21.15.163:0/1364347304 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fe74002feb0 0x7fe740032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:16.345 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.336+0000 7fe74affd700 1 -- 172.21.15.163:0/1364347304 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fe750040fe0 con 0x7fe75c0703f0 2023-11-15T04:55:16.345 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.337+0000 7fe759d9b700 1 --2- 172.21.15.163:0/1364347304 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fe74002feb0 0x7fe740032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:16.345 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.337+0000 7fe759d9b700 1 --2- 172.21.15.163:0/1364347304 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fe74002feb0 0x7fe740032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe74c0096a0 tx=0x7fe74c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:16.345 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.341+0000 7fe74affd700 1 -- 172.21.15.163:0/1364347304 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe75006d2b0 con 0x7fe75c0703f0 2023-11-15T04:55:16.519 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.518+0000 7fe7609a6700 1 -- 172.21.15.163:0/1364347304 --> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fe75c10beb0 con 0x7fe74002feb0 2023-11-15T04:55:16.523 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.522+0000 7fe74affd700 1 -- 172.21.15.163:0/1364347304 <== mgr.14142 v2:172.21.15.163:6800/2933442915 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fe75c10beb0 con 0x7fe74002feb0 2023-11-15T04:55:16.523 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-11-15T04:55:16.525 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.523+0000 7fe7609a6700 1 -- 172.21.15.163:0/1364347304 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fe74002feb0 msgr2=0x7fe740032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:16.526 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.523+0000 7fe7609a6700 1 --2- 172.21.15.163:0/1364347304 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fe74002feb0 0x7fe740032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe74c0096a0 tx=0x7fe74c006b40).stop 2023-11-15T04:55:16.526 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.523+0000 7fe7609a6700 1 -- 172.21.15.163:0/1364347304 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c0703f0 msgr2=0x7fe75c10aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:16.526 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.523+0000 7fe7609a6700 1 --2- 172.21.15.163:0/1364347304 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c0703f0 0x7fe75c10aad0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fe75000df90 tx=0x7fe750006ee0).stop 2023-11-15T04:55:16.526 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.523+0000 7fe7609a6700 1 -- 172.21.15.163:0/1364347304 shutdown_connections 2023-11-15T04:55:16.526 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.523+0000 7fe7609a6700 1 --2- 172.21.15.163:0/1364347304 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fe74002feb0 0x7fe740032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:16.527 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.523+0000 7fe7609a6700 1 --2- 172.21.15.163:0/1364347304 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c0703f0 0x7fe75c10aad0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:16.527 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.523+0000 7fe7609a6700 1 -- 172.21.15.163:0/1364347304 >> 172.21.15.163:0/1364347304 conn(0x7fe75c0f63c0 msgr2=0x7fe75c0f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:16.527 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.524+0000 7fe7609a6700 1 -- 172.21.15.163:0/1364347304 shutdown_connections 2023-11-15T04:55:16.527 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:16.524+0000 7fe7609a6700 1 -- 172.21.15.163:0/1364347304 wait complete. 2023-11-15T04:55:16.800 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:15.645478+0000 mgr.smithi163.qesyrx (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:55:16.801 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cephadm 2023-11-15T04:55:15.646095+0000 mgr.smithi163.qesyrx (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2023-11-15T04:55:16.801 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:16.523172+0000 mon.smithi163 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.163:0/2416123149' entity='mgr.smithi163.qesyrx' 2023-11-15T04:55:16.856 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2023-11-15T04:55:17.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.247+0000 7f05c87ef700 1 Processor -- start 2023-11-15T04:55:17.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.247+0000 7f05c87ef700 1 -- start start 2023-11-15T04:55:17.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.247+0000 7f05c87ef700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05c00703f0 0x7f05c00fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:17.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.247+0000 7f05c87ef700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c00fee10 con 0x7f05c00703f0 2023-11-15T04:55:17.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.247+0000 7f05c658b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05c00703f0 0x7f05c00fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:17.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.248+0000 7f05c658b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05c00703f0 0x7f05c00fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:46956/0 (socket says 172.21.15.163:46956) 2023-11-15T04:55:17.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.248+0000 7f05c658b700 1 -- 172.21.15.163:0/2956617581 learned_addr learned my addr 172.21.15.163:0/2956617581 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:17.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.248+0000 7f05c658b700 1 -- 172.21.15.163:0/2956617581 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c00fef50 con 0x7f05c00703f0 2023-11-15T04:55:17.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.248+0000 7f05c658b700 1 --2- 172.21.15.163:0/2956617581 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05c00703f0 0x7f05c00fe920 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f05b000b5f0 tx=0x7f05b0006ee0).ready entity=mon.0 client_cookie=7fffceb8e6bc3804 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:17.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.248+0000 7f05c5589700 1 -- 172.21.15.163:0/2956617581 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05b000d600 con 0x7f05c00703f0 2023-11-15T04:55:17.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.248+0000 7f05c5589700 1 -- 172.21.15.163:0/2956617581 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f05b000dc10 con 0x7f05c00703f0 2023-11-15T04:55:17.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.248+0000 7f05c5589700 1 -- 172.21.15.163:0/2956617581 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05b0011cd0 con 0x7f05c00703f0 2023-11-15T04:55:17.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.249+0000 7f05c87ef700 1 -- 172.21.15.163:0/2956617581 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05c00703f0 msgr2=0x7f05c00fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:17.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.249+0000 7f05c87ef700 1 --2- 172.21.15.163:0/2956617581 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05c00703f0 0x7f05c00fe920 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f05b000b5f0 tx=0x7f05b0006ee0).stop 2023-11-15T04:55:17.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.249+0000 7f05c87ef700 1 -- 172.21.15.163:0/2956617581 shutdown_connections 2023-11-15T04:55:17.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.249+0000 7f05c87ef700 1 --2- 172.21.15.163:0/2956617581 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05c00703f0 0x7f05c00fe920 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:17.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.249+0000 7f05c87ef700 1 -- 172.21.15.163:0/2956617581 >> 172.21.15.163:0/2956617581 conn(0x7f05c00f63c0 msgr2=0x7f05c00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:17.257 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.249+0000 7f05c87ef700 1 -- 172.21.15.163:0/2956617581 shutdown_connections 2023-11-15T04:55:17.257 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.249+0000 7f05c87ef700 1 -- 172.21.15.163:0/2956617581 wait complete. 2023-11-15T04:55:17.257 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.249+0000 7f05c87ef700 1 Processor -- start 2023-11-15T04:55:17.257 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.249+0000 7f05c87ef700 1 -- start start 2023-11-15T04:55:17.258 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.250+0000 7f05c87ef700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05c00703f0 0x7f05c010aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:17.258 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.250+0000 7f05c87ef700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c010af40 con 0x7f05c00703f0 2023-11-15T04:55:17.258 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.250+0000 7f05c658b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05c00703f0 0x7f05c010aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:17.258 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.250+0000 7f05c658b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05c00703f0 0x7f05c010aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:46966/0 (socket says 172.21.15.163:46966) 2023-11-15T04:55:17.258 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.250+0000 7f05c658b700 1 -- 172.21.15.163:0/2980265164 learned_addr learned my addr 172.21.15.163:0/2980265164 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:17.259 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.250+0000 7f05c658b700 1 -- 172.21.15.163:0/2980265164 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b000b200 con 0x7f05c00703f0 2023-11-15T04:55:17.259 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.250+0000 7f05c658b700 1 --2- 172.21.15.163:0/2980265164 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05c00703f0 0x7f05c010aa50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f05b0006a40 tx=0x7f05b0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:17.259 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.250+0000 7f05aeffd700 1 -- 172.21.15.163:0/2980265164 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05b0011c90 con 0x7f05c00703f0 2023-11-15T04:55:17.260 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.250+0000 7f05aeffd700 1 -- 172.21.15.163:0/2980265164 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f05b001b8c0 con 0x7f05c00703f0 2023-11-15T04:55:17.260 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.250+0000 7f05c87ef700 1 -- 172.21.15.163:0/2980265164 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c010b140 con 0x7f05c00703f0 2023-11-15T04:55:17.260 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.250+0000 7f05aeffd700 1 -- 172.21.15.163:0/2980265164 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05b001a8c0 con 0x7f05c00703f0 2023-11-15T04:55:17.260 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.250+0000 7f05c87ef700 1 -- 172.21.15.163:0/2980265164 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c010b550 con 0x7f05c00703f0 2023-11-15T04:55:17.260 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.251+0000 7f05aeffd700 1 -- 172.21.15.163:0/2980265164 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f05b000f070 con 0x7f05c00703f0 2023-11-15T04:55:17.260 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.251+0000 7f05aeffd700 1 --2- 172.21.15.163:0/2980265164 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f05a802fe60 0x7f05a8032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:17.261 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.251+0000 7f05aeffd700 1 -- 172.21.15.163:0/2980265164 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f05b00409d0 con 0x7f05c00703f0 2023-11-15T04:55:17.261 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.251+0000 7f05c87ef700 1 -- 172.21.15.163:0/2980265164 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c0045ad0 con 0x7f05c00703f0 2023-11-15T04:55:17.261 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.251+0000 7f05c5d8a700 1 --2- 172.21.15.163:0/2980265164 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f05a802fe60 0x7f05a8032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:17.261 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.252+0000 7f05c5d8a700 1 --2- 172.21.15.163:0/2980265164 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f05a802fe60 0x7f05a8032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f05bc0096a0 tx=0x7f05bc006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:17.262 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.256+0000 7f05aeffd700 1 -- 172.21.15.163:0/2980265164 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05b0016070 con 0x7f05c00703f0 2023-11-15T04:55:17.408 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.407+0000 7f05c87ef700 1 -- 172.21.15.163:0/2980265164 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f05c00ffdb0 con 0x7f05c00703f0 2023-11-15T04:55:17.411 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.410+0000 7f05aeffd700 1 -- 172.21.15.163:0/2980265164 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v9)=0 v9) v1 ==== 142+0+0 (secure 0 0 0) 0x7f05b0010930 con 0x7f05c00703f0 2023-11-15T04:55:17.414 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.412+0000 7f05c87ef700 1 -- 172.21.15.163:0/2980265164 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f05a802fe60 msgr2=0x7f05a8032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:17.414 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.412+0000 7f05c87ef700 1 --2- 172.21.15.163:0/2980265164 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f05a802fe60 0x7f05a8032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f05bc0096a0 tx=0x7f05bc006b40).stop 2023-11-15T04:55:17.414 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.412+0000 7f05c87ef700 1 -- 172.21.15.163:0/2980265164 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05c00703f0 msgr2=0x7f05c010aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:17.414 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.412+0000 7f05c87ef700 1 --2- 172.21.15.163:0/2980265164 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05c00703f0 0x7f05c010aa50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f05b0006a40 tx=0x7f05b0006ee0).stop 2023-11-15T04:55:17.415 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.412+0000 7f05c87ef700 1 -- 172.21.15.163:0/2980265164 shutdown_connections 2023-11-15T04:55:17.415 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.412+0000 7f05c87ef700 1 --2- 172.21.15.163:0/2980265164 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f05a802fe60 0x7f05a8032320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:17.415 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.412+0000 7f05c87ef700 1 --2- 172.21.15.163:0/2980265164 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05c00703f0 0x7f05c010aa50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:17.415 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.412+0000 7f05c87ef700 1 -- 172.21.15.163:0/2980265164 >> 172.21.15.163:0/2980265164 conn(0x7f05c00f63c0 msgr2=0x7f05c00f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:17.415 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.412+0000 7f05c87ef700 1 -- 172.21.15.163:0/2980265164 shutdown_connections 2023-11-15T04:55:17.416 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:17.412+0000 7f05c87ef700 1 -- 172.21.15.163:0/2980265164 wait complete. 2023-11-15T04:55:17.756 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2023-11-15T04:55:17.952 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:17 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:16.520366+0000 mgr.smithi163.qesyrx (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:55:17.953 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:17 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cephadm 2023-11-15T04:55:16.520876+0000 mgr.smithi163.qesyrx (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-11-15T04:55:17.953 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:17 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:17.411572+0000 mon.smithi163 (mon.0) 85 : audit [INF] from='client.? 172.21.15.163:0/2980265164' entity='client.admin' 2023-11-15T04:55:18.106 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.100+0000 7fd6c2ba2700 1 Processor -- start 2023-11-15T04:55:18.106 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.100+0000 7fd6c2ba2700 1 -- start start 2023-11-15T04:55:18.106 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.100+0000 7fd6c2ba2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6bc0fa9d0 0x7fd6bc0fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:18.106 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.100+0000 7fd6c2ba2700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc0fb2e0 con 0x7fd6bc0fa9d0 2023-11-15T04:55:18.107 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.100+0000 7fd6c1ba0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6bc0fa9d0 0x7fd6bc0fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:18.107 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.100+0000 7fd6c1ba0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6bc0fa9d0 0x7fd6bc0fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:46968/0 (socket says 172.21.15.163:46968) 2023-11-15T04:55:18.107 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.100+0000 7fd6c1ba0700 1 -- 172.21.15.163:0/81840525 learned_addr learned my addr 172.21.15.163:0/81840525 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:18.107 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.101+0000 7fd6c1ba0700 1 -- 172.21.15.163:0/81840525 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc0fb420 con 0x7fd6bc0fa9d0 2023-11-15T04:55:18.108 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.101+0000 7fd6c1ba0700 1 --2- 172.21.15.163:0/81840525 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6bc0fa9d0 0x7fd6bc0fadf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fd6b800b5f0 tx=0x7fd6b8006ee0).ready entity=mon.0 client_cookie=ecea908583c5dc3e server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:18.108 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.101+0000 7fd6c0b9e700 1 -- 172.21.15.163:0/81840525 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6b800d600 con 0x7fd6bc0fa9d0 2023-11-15T04:55:18.108 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.101+0000 7fd6c0b9e700 1 -- 172.21.15.163:0/81840525 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd6b800dc10 con 0x7fd6bc0fa9d0 2023-11-15T04:55:18.108 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.101+0000 7fd6c0b9e700 1 -- 172.21.15.163:0/81840525 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6b8011cd0 con 0x7fd6bc0fa9d0 2023-11-15T04:55:18.108 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.101+0000 7fd6c2ba2700 1 -- 172.21.15.163:0/81840525 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6bc0fa9d0 msgr2=0x7fd6bc0fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:18.109 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.101+0000 7fd6c2ba2700 1 --2- 172.21.15.163:0/81840525 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6bc0fa9d0 0x7fd6bc0fadf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fd6b800b5f0 tx=0x7fd6b8006ee0).stop 2023-11-15T04:55:18.109 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.102+0000 7fd6c2ba2700 1 -- 172.21.15.163:0/81840525 shutdown_connections 2023-11-15T04:55:18.109 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.102+0000 7fd6c2ba2700 1 --2- 172.21.15.163:0/81840525 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6bc0fa9d0 0x7fd6bc0fadf0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:18.109 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.102+0000 7fd6c2ba2700 1 -- 172.21.15.163:0/81840525 >> 172.21.15.163:0/81840525 conn(0x7fd6bc0f6210 msgr2=0x7fd6bc0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:18.109 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.102+0000 7fd6c2ba2700 1 -- 172.21.15.163:0/81840525 shutdown_connections 2023-11-15T04:55:18.110 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.102+0000 7fd6c2ba2700 1 -- 172.21.15.163:0/81840525 wait complete. 2023-11-15T04:55:18.110 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.102+0000 7fd6c2ba2700 1 Processor -- start 2023-11-15T04:55:18.110 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.102+0000 7fd6c2ba2700 1 -- start start 2023-11-15T04:55:18.110 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.102+0000 7fd6c2ba2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6bc0fa9d0 0x7fd6bc10a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:18.110 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.102+0000 7fd6c2ba2700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc10ad50 con 0x7fd6bc0fa9d0 2023-11-15T04:55:18.111 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.102+0000 7fd6c1ba0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6bc0fa9d0 0x7fd6bc10a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:18.111 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.102+0000 7fd6c1ba0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6bc0fa9d0 0x7fd6bc10a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:46984/0 (socket says 172.21.15.163:46984) 2023-11-15T04:55:18.111 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.102+0000 7fd6c1ba0700 1 -- 172.21.15.163:0/989534207 learned_addr learned my addr 172.21.15.163:0/989534207 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:18.111 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.103+0000 7fd6c1ba0700 1 -- 172.21.15.163:0/989534207 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b800b200 con 0x7fd6bc0fa9d0 2023-11-15T04:55:18.111 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.103+0000 7fd6c1ba0700 1 --2- 172.21.15.163:0/989534207 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6bc0fa9d0 0x7fd6bc10a860 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd6b800e040 tx=0x7fd6b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:18.112 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.103+0000 7fd6aa7fc700 1 -- 172.21.15.163:0/989534207 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6b801b410 con 0x7fd6bc0fa9d0 2023-11-15T04:55:18.112 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.103+0000 7fd6c2ba2700 1 -- 172.21.15.163:0/989534207 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6bc10af50 con 0x7fd6bc0fa9d0 2023-11-15T04:55:18.112 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.103+0000 7fd6aa7fc700 1 -- 172.21.15.163:0/989534207 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd6b801ba20 con 0x7fd6bc0fa9d0 2023-11-15T04:55:18.112 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.103+0000 7fd6aa7fc700 1 -- 172.21.15.163:0/989534207 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6b801a910 con 0x7fd6bc0fa9d0 2023-11-15T04:55:18.112 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.103+0000 7fd6c2ba2700 1 -- 172.21.15.163:0/989534207 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6bc10b360 con 0x7fd6bc0fa9d0 2023-11-15T04:55:18.113 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.104+0000 7fd6aa7fc700 1 -- 172.21.15.163:0/989534207 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fd6b800f070 con 0x7fd6bc0fa9d0 2023-11-15T04:55:18.113 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.104+0000 7fd6aa7fc700 1 --2- 172.21.15.163:0/989534207 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fd6a402fe60 0x7fd6a4032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:18.113 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.104+0000 7fd6c2ba2700 1 -- 172.21.15.163:0/989534207 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6bc045ad0 con 0x7fd6bc0fa9d0 2023-11-15T04:55:18.113 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.104+0000 7fd6aa7fc700 1 -- 172.21.15.163:0/989534207 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fd6b8041270 con 0x7fd6bc0fa9d0 2023-11-15T04:55:18.113 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.104+0000 7fd6c139f700 1 --2- 172.21.15.163:0/989534207 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fd6a402fe60 0x7fd6a4032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:18.114 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.104+0000 7fd6c139f700 1 --2- 172.21.15.163:0/989534207 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fd6a402fe60 0x7fd6a4032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fd6ac0096a0 tx=0x7fd6ac006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:18.114 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.108+0000 7fd6aa7fc700 1 -- 172.21.15.163:0/989534207 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd6b8016080 con 0x7fd6bc0fa9d0 2023-11-15T04:55:18.259 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.258+0000 7fd6c2ba2700 1 -- 172.21.15.163:0/989534207 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fd6bc0fc750 con 0x7fd6bc0fa9d0 2023-11-15T04:55:18.262 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.261+0000 7fd6aa7fc700 1 -- 172.21.15.163:0/989534207 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7fd6b801ab80 con 0x7fd6bc0fa9d0 2023-11-15T04:55:18.265 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.262+0000 7fd6c2ba2700 1 -- 172.21.15.163:0/989534207 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fd6a402fe60 msgr2=0x7fd6a4032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:18.265 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.262+0000 7fd6c2ba2700 1 --2- 172.21.15.163:0/989534207 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fd6a402fe60 0x7fd6a4032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fd6ac0096a0 tx=0x7fd6ac006b40).stop 2023-11-15T04:55:18.265 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.262+0000 7fd6c2ba2700 1 -- 172.21.15.163:0/989534207 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6bc0fa9d0 msgr2=0x7fd6bc10a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:18.265 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.262+0000 7fd6c2ba2700 1 --2- 172.21.15.163:0/989534207 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6bc0fa9d0 0x7fd6bc10a860 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd6b800e040 tx=0x7fd6b8006ee0).stop 2023-11-15T04:55:18.266 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.262+0000 7fd6c2ba2700 1 -- 172.21.15.163:0/989534207 shutdown_connections 2023-11-15T04:55:18.266 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.262+0000 7fd6c2ba2700 1 --2- 172.21.15.163:0/989534207 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fd6a402fe60 0x7fd6a4032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:18.266 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.262+0000 7fd6c2ba2700 1 --2- 172.21.15.163:0/989534207 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6bc0fa9d0 0x7fd6bc10a860 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:18.266 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.262+0000 7fd6c2ba2700 1 -- 172.21.15.163:0/989534207 >> 172.21.15.163:0/989534207 conn(0x7fd6bc0f6210 msgr2=0x7fd6bc0f7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:18.267 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.262+0000 7fd6c2ba2700 1 -- 172.21.15.163:0/989534207 shutdown_connections 2023-11-15T04:55:18.267 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:18.263+0000 7fd6c2ba2700 1 -- 172.21.15.163:0/989534207 wait complete. 2023-11-15T04:55:18.647 INFO:teuthology.orchestra.run.smithi163.stderr:Enabling the dashboard module... 2023-11-15T04:55:18.648 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2023-11-15T04:55:19.005 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.002+0000 7f9852d5b700 1 Processor -- start 2023-11-15T04:55:19.005 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.002+0000 7f9852d5b700 1 -- start start 2023-11-15T04:55:19.006 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.002+0000 7f9852d5b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f984c070370 0x7f984c0fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:19.006 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.002+0000 7f9852d5b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984c070790 con 0x7f984c070370 2023-11-15T04:55:19.006 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.002+0000 7f9850af7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f984c070370 0x7f984c0fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:19.006 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.002+0000 7f9850af7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f984c070370 0x7f984c0fe9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:46998/0 (socket says 172.21.15.163:46998) 2023-11-15T04:55:19.006 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.002+0000 7f9850af7700 1 -- 172.21.15.163:0/2378249956 learned_addr learned my addr 172.21.15.163:0/2378249956 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:19.007 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.002+0000 7f9850af7700 1 -- 172.21.15.163:0/2378249956 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984c0fee90 con 0x7f984c070370 2023-11-15T04:55:19.007 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.002+0000 7f9850af7700 1 --2- 172.21.15.163:0/2378249956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f984c070370 0x7f984c0fe9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f984000b5f0 tx=0x7f9840006ee0).ready entity=mon.0 client_cookie=553a31bd6a47f90 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:19.007 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.003+0000 7f984b7fe700 1 -- 172.21.15.163:0/2378249956 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f984000d600 con 0x7f984c070370 2023-11-15T04:55:19.007 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.003+0000 7f984b7fe700 1 -- 172.21.15.163:0/2378249956 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f984000dc10 con 0x7f984c070370 2023-11-15T04:55:19.008 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.003+0000 7f984b7fe700 1 -- 172.21.15.163:0/2378249956 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9840011cd0 con 0x7f984c070370 2023-11-15T04:55:19.008 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.003+0000 7f9852d5b700 1 -- 172.21.15.163:0/2378249956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f984c070370 msgr2=0x7f984c0fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:19.008 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.003+0000 7f9852d5b700 1 --2- 172.21.15.163:0/2378249956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f984c070370 0x7f984c0fe9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f984000b5f0 tx=0x7f9840006ee0).stop 2023-11-15T04:55:19.008 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.003+0000 7f9852d5b700 1 -- 172.21.15.163:0/2378249956 shutdown_connections 2023-11-15T04:55:19.008 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.003+0000 7f9852d5b700 1 --2- 172.21.15.163:0/2378249956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f984c070370 0x7f984c0fe9a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:19.009 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.003+0000 7f9852d5b700 1 -- 172.21.15.163:0/2378249956 >> 172.21.15.163:0/2378249956 conn(0x7f984c0f63c0 msgr2=0x7f984c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:19.009 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.003+0000 7f9852d5b700 1 -- 172.21.15.163:0/2378249956 shutdown_connections 2023-11-15T04:55:19.010 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.003+0000 7f9852d5b700 1 -- 172.21.15.163:0/2378249956 wait complete. 2023-11-15T04:55:19.011 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.004+0000 7f9852d5b700 1 Processor -- start 2023-11-15T04:55:19.011 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.004+0000 7f9852d5b700 1 -- start start 2023-11-15T04:55:19.013 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.004+0000 7f9852d5b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f984c070370 0x7f984c0fc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:19.014 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.004+0000 7f9852d5b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984c0fa8a0 con 0x7f984c070370 2023-11-15T04:55:19.014 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.004+0000 7f9850af7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f984c070370 0x7f984c0fc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:19.014 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.004+0000 7f9850af7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f984c070370 0x7f984c0fc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:47008/0 (socket says 172.21.15.163:47008) 2023-11-15T04:55:19.014 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.004+0000 7f9850af7700 1 -- 172.21.15.163:0/2092650542 learned_addr learned my addr 172.21.15.163:0/2092650542 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:19.014 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.005+0000 7f9850af7700 1 -- 172.21.15.163:0/2092650542 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984000b200 con 0x7f984c070370 2023-11-15T04:55:19.015 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.005+0000 7f9850af7700 1 --2- 172.21.15.163:0/2092650542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f984c070370 0x7f984c0fc420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f9840006800 tx=0x7f9840006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:19.016 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.005+0000 7f98497fa700 1 -- 172.21.15.163:0/2092650542 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9840011df0 con 0x7f984c070370 2023-11-15T04:55:19.016 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.005+0000 7f98497fa700 1 -- 172.21.15.163:0/2092650542 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f984001b8c0 con 0x7f984c070370 2023-11-15T04:55:19.016 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.005+0000 7f98497fa700 1 -- 172.21.15.163:0/2092650542 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f984001a890 con 0x7f984c070370 2023-11-15T04:55:19.017 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.005+0000 7f9852d5b700 1 -- 172.21.15.163:0/2092650542 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f984c0faa40 con 0x7f984c070370 2023-11-15T04:55:19.017 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.005+0000 7f9852d5b700 1 -- 172.21.15.163:0/2092650542 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f984c0fae50 con 0x7f984c070370 2023-11-15T04:55:19.017 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.006+0000 7f98497fa700 1 -- 172.21.15.163:0/2092650542 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f984000f070 con 0x7f984c070370 2023-11-15T04:55:19.017 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.006+0000 7f98497fa700 1 --2- 172.21.15.163:0/2092650542 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f983002feb0 0x7f9830032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:19.018 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.006+0000 7f98497fa700 1 -- 172.21.15.163:0/2092650542 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f9840040ff0 con 0x7f984c070370 2023-11-15T04:55:19.018 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.006+0000 7f984bfff700 1 --2- 172.21.15.163:0/2092650542 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f983002feb0 0x7f9830032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:19.018 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.006+0000 7f9852d5b700 1 -- 172.21.15.163:0/2092650542 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f984c059100 con 0x7f984c070370 2023-11-15T04:55:19.018 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.006+0000 7f984bfff700 1 --2- 172.21.15.163:0/2092650542 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f983002feb0 0x7f9830032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f98380096a0 tx=0x7f9838006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:19.019 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.010+0000 7f98497fa700 1 -- 172.21.15.163:0/2092650542 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9840016070 con 0x7f984c070370 2023-11-15T04:55:19.200 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.198+0000 7f9852d5b700 1 -- 172.21.15.163:0/2092650542 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f984c045ad0 con 0x7f984c070370 2023-11-15T04:55:19.262 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.260+0000 7f98497fa700 1 -- 172.21.15.163:0/2092650542 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f9840024790 con 0x7f984c070370 2023-11-15T04:55:19.263 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.262+0000 7f98497fa700 1 -- 172.21.15.163:0/2092650542 <== mon.0 v2:172.21.15.163:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v12) v1 ==== 88+0+0 (secure 0 0 0) 0x7f984002d6c0 con 0x7f984c070370 2023-11-15T04:55:19.266 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.264+0000 7f9852d5b700 1 -- 172.21.15.163:0/2092650542 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f983002feb0 msgr2=0x7f9830032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:19.266 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.264+0000 7f9852d5b700 1 --2- 172.21.15.163:0/2092650542 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f983002feb0 0x7f9830032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f98380096a0 tx=0x7f9838006b40).stop 2023-11-15T04:55:19.266 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.264+0000 7f9852d5b700 1 -- 172.21.15.163:0/2092650542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f984c070370 msgr2=0x7f984c0fc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:19.266 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.264+0000 7f9852d5b700 1 --2- 172.21.15.163:0/2092650542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f984c070370 0x7f984c0fc420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f9840006800 tx=0x7f9840006ee0).stop 2023-11-15T04:55:19.267 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.264+0000 7f9852d5b700 1 -- 172.21.15.163:0/2092650542 shutdown_connections 2023-11-15T04:55:19.267 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.264+0000 7f9852d5b700 1 --2- 172.21.15.163:0/2092650542 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f983002feb0 0x7f9830032370 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:19.267 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.264+0000 7f9852d5b700 1 --2- 172.21.15.163:0/2092650542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f984c070370 0x7f984c0fc420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:19.267 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.264+0000 7f9852d5b700 1 -- 172.21.15.163:0/2092650542 >> 172.21.15.163:0/2092650542 conn(0x7f984c0f63c0 msgr2=0x7f984c0f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:19.267 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.264+0000 7f9852d5b700 1 -- 172.21.15.163:0/2092650542 shutdown_connections 2023-11-15T04:55:19.268 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:19.264+0000 7f9852d5b700 1 -- 172.21.15.163:0/2092650542 wait complete. 2023-11-15T04:55:19.575 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:19 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:18.262322+0000 mon.smithi163 (mon.0) 86 : audit [INF] from='client.? 172.21.15.163:0/989534207' entity='client.admin' 2023-11-15T04:55:19.575 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:19 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:19.200632+0000 mon.smithi163 (mon.0) 87 : audit [INF] from='client.? 172.21.15.163:0/2092650542' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-11-15T04:55:19.605 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-15T04:55:20.167 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.161+0000 7fa766930700 1 Processor -- start 2023-11-15T04:55:20.167 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.161+0000 7fa766930700 1 -- start start 2023-11-15T04:55:20.167 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.161+0000 7fa766930700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7600fd380 0x7fa7600fd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:20.168 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.161+0000 7fa766930700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7600fdc90 con 0x7fa7600fd380 2023-11-15T04:55:20.168 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.162+0000 7fa75ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7600fd380 0x7fa7600fd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:20.168 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.162+0000 7fa75ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7600fd380 0x7fa7600fd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:47044/0 (socket says 172.21.15.163:47044) 2023-11-15T04:55:20.168 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.162+0000 7fa75ffff700 1 -- 172.21.15.163:0/1682913074 learned_addr learned my addr 172.21.15.163:0/1682913074 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:20.169 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.162+0000 7fa75ffff700 1 -- 172.21.15.163:0/1682913074 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7600fddd0 con 0x7fa7600fd380 2023-11-15T04:55:20.169 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.162+0000 7fa75ffff700 1 --2- 172.21.15.163:0/1682913074 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7600fd380 0x7fa7600fd7a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fa75000b5f0 tx=0x7fa750006ee0).ready entity=mon.0 client_cookie=de2d4543e38b460b server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:20.169 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.162+0000 7fa75effd700 1 -- 172.21.15.163:0/1682913074 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa75000d600 con 0x7fa7600fd380 2023-11-15T04:55:20.169 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.162+0000 7fa75effd700 1 -- 172.21.15.163:0/1682913074 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa75000dc10 con 0x7fa7600fd380 2023-11-15T04:55:20.169 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.162+0000 7fa75effd700 1 -- 172.21.15.163:0/1682913074 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa750011cd0 con 0x7fa7600fd380 2023-11-15T04:55:20.170 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.163+0000 7fa766930700 1 -- 172.21.15.163:0/1682913074 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7600fd380 msgr2=0x7fa7600fd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:20.170 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.163+0000 7fa766930700 1 --2- 172.21.15.163:0/1682913074 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7600fd380 0x7fa7600fd7a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fa75000b5f0 tx=0x7fa750006ee0).stop 2023-11-15T04:55:20.170 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.163+0000 7fa766930700 1 -- 172.21.15.163:0/1682913074 shutdown_connections 2023-11-15T04:55:20.171 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.163+0000 7fa766930700 1 --2- 172.21.15.163:0/1682913074 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7600fd380 0x7fa7600fd7a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:20.171 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.163+0000 7fa766930700 1 -- 172.21.15.163:0/1682913074 >> 172.21.15.163:0/1682913074 conn(0x7fa7600f5b40 msgr2=0x7fa7600f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:20.171 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.163+0000 7fa766930700 1 -- 172.21.15.163:0/1682913074 shutdown_connections 2023-11-15T04:55:20.172 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.163+0000 7fa766930700 1 -- 172.21.15.163:0/1682913074 wait complete. 2023-11-15T04:55:20.172 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.164+0000 7fa766930700 1 Processor -- start 2023-11-15T04:55:20.172 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.164+0000 7fa766930700 1 -- start start 2023-11-15T04:55:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.164+0000 7fa766930700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7600fd380 0x7fa76010aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.164+0000 7fa766930700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76010af10 con 0x7fa7600fd380 2023-11-15T04:55:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.164+0000 7fa75ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7600fd380 0x7fa76010aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.165+0000 7fa75ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7600fd380 0x7fa76010aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:47054/0 (socket says 172.21.15.163:47054) 2023-11-15T04:55:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.165+0000 7fa75ffff700 1 -- 172.21.15.163:0/4283785688 learned_addr learned my addr 172.21.15.163:0/4283785688 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:20.174 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.165+0000 7fa75ffff700 1 -- 172.21.15.163:0/4283785688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa75000b200 con 0x7fa7600fd380 2023-11-15T04:55:20.174 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.165+0000 7fa75ffff700 1 --2- 172.21.15.163:0/4283785688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7600fd380 0x7fa76010aa20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fa75000e040 tx=0x7fa750006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:20.174 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.165+0000 7fa75cff9700 1 -- 172.21.15.163:0/4283785688 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa75001b410 con 0x7fa7600fd380 2023-11-15T04:55:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.165+0000 7fa75cff9700 1 -- 172.21.15.163:0/4283785688 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa75001ba20 con 0x7fa7600fd380 2023-11-15T04:55:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.165+0000 7fa766930700 1 -- 172.21.15.163:0/4283785688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa76010b110 con 0x7fa7600fd380 2023-11-15T04:55:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.165+0000 7fa75cff9700 1 -- 172.21.15.163:0/4283785688 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa75001a910 con 0x7fa7600fd380 2023-11-15T04:55:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.165+0000 7fa766930700 1 -- 172.21.15.163:0/4283785688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa76010b520 con 0x7fa7600fd380 2023-11-15T04:55:20.176 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.166+0000 7fa75cff9700 1 -- 172.21.15.163:0/4283785688 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fa75000f070 con 0x7fa7600fd380 2023-11-15T04:55:20.177 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.166+0000 7fa75cff9700 1 --2- 172.21.15.163:0/4283785688 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fa74c02ff00 0x7fa74c0323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:20.177 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.166+0000 7fa75cff9700 1 -- 172.21.15.163:0/4283785688 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fa750041150 con 0x7fa7600fd380 2023-11-15T04:55:20.177 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.166+0000 7fa75f7fe700 1 -- 172.21.15.163:0/4283785688 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fa74c02ff00 msgr2=0x7fa74c0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.163:6800/2933442915 2023-11-15T04:55:20.177 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.166+0000 7fa75f7fe700 1 --2- 172.21.15.163:0/4283785688 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fa74c02ff00 0x7fa74c0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T04:55:20.177 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.166+0000 7fa766930700 1 -- 172.21.15.163:0/4283785688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa760059100 con 0x7fa7600fd380 2023-11-15T04:55:20.178 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.171+0000 7fa75cff9700 1 -- 172.21.15.163:0/4283785688 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa750016070 con 0x7fa7600fd380 2023-11-15T04:55:20.362 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.360+0000 7fa766930700 1 -- 172.21.15.163:0/4283785688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa7600fa700 con 0x7fa7600fd380 2023-11-15T04:55:20.363 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.360+0000 7fa75cff9700 1 -- 172.21.15.163:0/4283785688 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v12) v1 ==== 56+0+104 (secure 0 0 0) 0x7fa750024410 con 0x7fa7600fd380 2023-11-15T04:55:20.363 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: { 2023-11-15T04:55:20.363 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 12, 2023-11-15T04:55:20.363 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "available": true, 2023-11-15T04:55:20.363 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "active_name": "smithi163.qesyrx", 2023-11-15T04:55:20.364 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-15T04:55:20.364 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: } 2023-11-15T04:55:20.365 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.363+0000 7fa766930700 1 -- 172.21.15.163:0/4283785688 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fa74c02ff00 msgr2=0x7fa74c0323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T04:55:20.365 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.363+0000 7fa766930700 1 --2- 172.21.15.163:0/4283785688 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fa74c02ff00 0x7fa74c0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:55:20.366 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.363+0000 7fa766930700 1 -- 172.21.15.163:0/4283785688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7600fd380 msgr2=0x7fa76010aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:20.366 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.363+0000 7fa766930700 1 --2- 172.21.15.163:0/4283785688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7600fd380 0x7fa76010aa20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fa75000e040 tx=0x7fa750006ee0).stop 2023-11-15T04:55:20.366 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.363+0000 7fa766930700 1 -- 172.21.15.163:0/4283785688 shutdown_connections 2023-11-15T04:55:20.366 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.363+0000 7fa766930700 1 --2- 172.21.15.163:0/4283785688 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7fa74c02ff00 0x7fa74c0323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:55:20.367 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.363+0000 7fa766930700 1 --2- 172.21.15.163:0/4283785688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7600fd380 0x7fa76010aa20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:20.367 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.363+0000 7fa766930700 1 -- 172.21.15.163:0/4283785688 >> 172.21.15.163:0/4283785688 conn(0x7fa7600f5b40 msgr2=0x7fa7600ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:20.367 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.363+0000 7fa766930700 1 -- 172.21.15.163:0/4283785688 shutdown_connections 2023-11-15T04:55:20.367 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:20.363+0000 7fa766930700 1 -- 172.21.15.163:0/4283785688 wait complete. 2023-11-15T04:55:20.561 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:20 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:19.263488+0000 mon.smithi163 (mon.0) 88 : audit [INF] from='client.? 172.21.15.163:0/2092650542' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-11-15T04:55:20.562 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:20 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:19.263536+0000 mon.smithi163 (mon.0) 89 : cluster [DBG] mgrmap e12: smithi163.qesyrx(active, since 7s) 2023-11-15T04:55:20.706 INFO:teuthology.orchestra.run.smithi163.stderr:Waiting for the mgr to restart... 2023-11-15T04:55:20.706 INFO:teuthology.orchestra.run.smithi163.stderr:Waiting for mgr epoch 12... 2023-11-15T04:55:20.707 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-15T04:55:21.076 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.074+0000 7f417647d700 1 Processor -- start 2023-11-15T04:55:21.077 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.074+0000 7f417647d700 1 -- start start 2023-11-15T04:55:21.077 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.074+0000 7f417647d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4170070430 0x7f41700fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:21.077 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.074+0000 7f417647d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41700fedd0 con 0x7f4170070430 2023-11-15T04:55:21.077 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.074+0000 7f416ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4170070430 0x7f41700fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:21.078 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.074+0000 7f416ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4170070430 0x7f41700fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:47068/0 (socket says 172.21.15.163:47068) 2023-11-15T04:55:21.078 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.074+0000 7f416ffff700 1 -- 172.21.15.163:0/1788243530 learned_addr learned my addr 172.21.15.163:0/1788243530 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:21.078 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.075+0000 7f416ffff700 1 -- 172.21.15.163:0/1788243530 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41700fef10 con 0x7f4170070430 2023-11-15T04:55:21.078 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.075+0000 7f416ffff700 1 --2- 172.21.15.163:0/1788243530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4170070430 0x7f41700fe8e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f416000b5f0 tx=0x7f4160006ee0).ready entity=mon.0 client_cookie=af4a950655d42860 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:21.078 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.075+0000 7f416effd700 1 -- 172.21.15.163:0/1788243530 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f416000d600 con 0x7f4170070430 2023-11-15T04:55:21.079 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.075+0000 7f416effd700 1 -- 172.21.15.163:0/1788243530 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f416000dc10 con 0x7f4170070430 2023-11-15T04:55:21.079 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.075+0000 7f416effd700 1 -- 172.21.15.163:0/1788243530 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4160011cd0 con 0x7f4170070430 2023-11-15T04:55:21.081 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.076+0000 7f417647d700 1 -- 172.21.15.163:0/1788243530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4170070430 msgr2=0x7f41700fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:21.081 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.076+0000 7f417647d700 1 --2- 172.21.15.163:0/1788243530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4170070430 0x7f41700fe8e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f416000b5f0 tx=0x7f4160006ee0).stop 2023-11-15T04:55:21.082 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.076+0000 7f417647d700 1 -- 172.21.15.163:0/1788243530 shutdown_connections 2023-11-15T04:55:21.082 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.076+0000 7f417647d700 1 --2- 172.21.15.163:0/1788243530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4170070430 0x7f41700fe8e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:21.082 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.076+0000 7f417647d700 1 -- 172.21.15.163:0/1788243530 >> 172.21.15.163:0/1788243530 conn(0x7f41700f63c0 msgr2=0x7f41700f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:21.082 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.076+0000 7f417647d700 1 -- 172.21.15.163:0/1788243530 shutdown_connections 2023-11-15T04:55:21.083 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.076+0000 7f417647d700 1 -- 172.21.15.163:0/1788243530 wait complete. 2023-11-15T04:55:21.083 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.077+0000 7f417647d700 1 Processor -- start 2023-11-15T04:55:21.083 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.077+0000 7f417647d700 1 -- start start 2023-11-15T04:55:21.083 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.077+0000 7f417647d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4170070430 0x7f417010ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:21.083 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.077+0000 7f417647d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417010aff0 con 0x7f4170070430 2023-11-15T04:55:21.083 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.077+0000 7f416ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4170070430 0x7f417010ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:21.084 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.077+0000 7f416ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4170070430 0x7f417010ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:47084/0 (socket says 172.21.15.163:47084) 2023-11-15T04:55:21.084 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.077+0000 7f416ffff700 1 -- 172.21.15.163:0/2967860199 learned_addr learned my addr 172.21.15.163:0/2967860199 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:21.084 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.077+0000 7f416ffff700 1 -- 172.21.15.163:0/2967860199 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416000b200 con 0x7f4170070430 2023-11-15T04:55:21.084 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.078+0000 7f416ffff700 1 --2- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4170070430 0x7f417010ab00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f4160006800 tx=0x7f4160006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:21.084 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.078+0000 7f416cff9700 1 -- 172.21.15.163:0/2967860199 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4160011df0 con 0x7f4170070430 2023-11-15T04:55:21.085 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.078+0000 7f417647d700 1 -- 172.21.15.163:0/2967860199 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f417010b1f0 con 0x7f4170070430 2023-11-15T04:55:21.085 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.078+0000 7f416cff9700 1 -- 172.21.15.163:0/2967860199 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f416001b8c0 con 0x7f4170070430 2023-11-15T04:55:21.085 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.078+0000 7f416cff9700 1 -- 172.21.15.163:0/2967860199 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f416001a830 con 0x7f4170070430 2023-11-15T04:55:21.085 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.078+0000 7f417647d700 1 -- 172.21.15.163:0/2967860199 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f417010b600 con 0x7f4170070430 2023-11-15T04:55:21.085 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.079+0000 7f416cff9700 1 -- 172.21.15.163:0/2967860199 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f416000f070 con 0x7f4170070430 2023-11-15T04:55:21.086 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.079+0000 7f416cff9700 1 --2- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f415c02feb0 0x7f415c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:21.086 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.079+0000 7f416cff9700 1 -- 172.21.15.163:0/2967860199 --> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f415c032a30 con 0x7f415c02feb0 2023-11-15T04:55:21.086 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.079+0000 7f416cff9700 1 -- 172.21.15.163:0/2967860199 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f4160041060 con 0x7f4170070430 2023-11-15T04:55:21.086 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.079+0000 7f416f7fe700 1 -- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f415c02feb0 msgr2=0x7f415c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.163:6800/2933442915 2023-11-15T04:55:21.086 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.079+0000 7f416f7fe700 1 --2- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f415c02feb0 0x7f415c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T04:55:21.281 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.280+0000 7f416f7fe700 1 -- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f415c02feb0 msgr2=0x7f415c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.163:6800/2933442915 2023-11-15T04:55:21.281 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.280+0000 7f416f7fe700 1 --2- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f415c02feb0 0x7f415c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T04:55:21.682 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.681+0000 7f416f7fe700 1 -- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f415c02feb0 msgr2=0x7f415c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.163:6800/2933442915 2023-11-15T04:55:21.682 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:21.681+0000 7f416f7fe700 1 --2- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f415c02feb0 0x7f415c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T04:55:21.702 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:21 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:20.362178+0000 mon.smithi163 (mon.0) 90 : audit [DBG] from='client.? 172.21.15.163:0/4283785688' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-15T04:55:22.483 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:22.482+0000 7f416f7fe700 1 -- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f415c02feb0 msgr2=0x7f415c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.163:6800/2933442915 2023-11-15T04:55:22.484 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:22.482+0000 7f416f7fe700 1 --2- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f415c02feb0 0x7f415c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T04:55:24.085 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:24.083+0000 7f416f7fe700 1 -- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f415c02feb0 msgr2=0x7f415c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.163:6800/2933442915 2023-11-15T04:55:24.085 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:24.083+0000 7f416f7fe700 1 --2- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f415c02feb0 0x7f415c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-15T04:55:24.306 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:24.304+0000 7f416cff9700 1 -- 172.21.15.163:0/2967860199 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mgrmap(e 13) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f416001ad50 con 0x7f4170070430 2023-11-15T04:55:24.306 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:24.304+0000 7f416cff9700 1 -- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f415c02feb0 msgr2=0x7f415c032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T04:55:24.306 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:24.304+0000 7f416cff9700 1 --2- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f415c02feb0 0x7f415c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:55:24.538 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:24 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:24.253180+0000 mon.smithi163 (mon.0) 91 : cluster [INF] Active manager daemon smithi163.qesyrx restarted 2023-11-15T04:55:24.538 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:24 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:24.253588+0000 mon.smithi163 (mon.0) 92 : cluster [INF] Activating manager daemon smithi163.qesyrx 2023-11-15T04:55:24.538 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:24 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:24.257304+0000 mon.smithi163 (mon.0) 93 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-11-15T04:55:25.313 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:25.310+0000 7f416cff9700 1 -- 172.21.15.163:0/2967860199 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f4160042500 con 0x7f4170070430 2023-11-15T04:55:25.314 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:25.310+0000 7f416cff9700 1 --2- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f415c033b40 0x7f415c035f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:25.314 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:25.310+0000 7f416cff9700 1 -- 172.21.15.163:0/2967860199 --> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f415c032a30 con 0x7f415c033b40 2023-11-15T04:55:25.314 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:25.311+0000 7f416f7fe700 1 --2- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f415c033b40 0x7f415c035f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:25.314 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:25.311+0000 7f416f7fe700 1 --2- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f415c033b40 0x7f415c035f30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f4164006fd0 tx=0x7f4164006cf0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:25.315 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:25.312+0000 7f416cff9700 1 -- 172.21.15.163:0/2967860199 <== mgr.14162 v2:172.21.15.163:6800/3399151648 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f415c032a30 con 0x7f415c033b40 2023-11-15T04:55:25.321 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:25.319+0000 7f417647d700 1 -- 172.21.15.163:0/2967860199 --> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f4170059100 con 0x7f415c033b40 2023-11-15T04:55:25.322 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:25.319+0000 7f416cff9700 1 -- 172.21.15.163:0/2967860199 <== mgr.14162 v2:172.21.15.163:6800/3399151648 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f4170059100 con 0x7f415c033b40 2023-11-15T04:55:25.322 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: { 2023-11-15T04:55:25.322 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "mgrmap_epoch": 14, 2023-11-15T04:55:25.322 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "initialized": true 2023-11-15T04:55:25.322 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: } 2023-11-15T04:55:25.323 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:25.319+0000 7f417647d700 1 -- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f415c033b40 msgr2=0x7f415c035f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:25.323 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:25.319+0000 7f417647d700 1 --2- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f415c033b40 0x7f415c035f30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f4164006fd0 tx=0x7f4164006cf0).stop 2023-11-15T04:55:25.323 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:25.319+0000 7f417647d700 1 -- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4170070430 msgr2=0x7f417010ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:25.323 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:25.319+0000 7f417647d700 1 --2- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4170070430 0x7f417010ab00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f4160006800 tx=0x7f4160006ee0).stop 2023-11-15T04:55:25.323 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:25.319+0000 7f417647d700 1 -- 172.21.15.163:0/2967860199 shutdown_connections 2023-11-15T04:55:25.324 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:25.319+0000 7f417647d700 1 --2- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f415c033b40 0x7f415c035f30 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:25.324 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:25.319+0000 7f417647d700 1 --2- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4170070430 0x7f417010ab00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:25.324 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:25.319+0000 7f417647d700 1 --2- 172.21.15.163:0/2967860199 >> [v2:172.21.15.163:6800/2933442915,v1:172.21.15.163:6801/2933442915] conn(0x7f415c02feb0 0x7f415c032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T04:55:25.324 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:25.319+0000 7f417647d700 1 -- 172.21.15.163:0/2967860199 >> 172.21.15.163:0/2967860199 conn(0x7f41700f63c0 msgr2=0x7f41700f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:25.324 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:25.320+0000 7f417647d700 1 -- 172.21.15.163:0/2967860199 shutdown_connections 2023-11-15T04:55:25.325 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:25.320+0000 7f417647d700 1 -- 172.21.15.163:0/2967860199 wait complete. 2023-11-15T04:55:25.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:24.306697+0000 mon.smithi163 (mon.0) 94 : cluster [DBG] mgrmap e13: smithi163.qesyrx(active, starting, since 0.0532166s) 2023-11-15T04:55:25.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:24.307782+0000 mon.smithi163 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2023-11-15T04:55:25.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:24.308360+0000 mon.smithi163 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "mgr metadata", "who": "smithi163.qesyrx", "id": "smithi163.qesyrx"}]: dispatch 2023-11-15T04:55:25.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:24.309256+0000 mon.smithi163 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T04:55:25.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:24.309429+0000 mon.smithi163 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T04:55:25.454 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:24.309560+0000 mon.smithi163 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T04:55:25.454 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:24.322991+0000 mon.smithi163 (mon.0) 100 : cluster [INF] Manager daemon smithi163.qesyrx is now available 2023-11-15T04:55:25.454 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:24.421965+0000 mon.smithi163 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:55:25.454 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:24.424096+0000 mon.smithi163 (mon.0) 102 : audit [DBG] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T04:55:25.454 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:24.445993+0000 mon.smithi163 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:55:25.455 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:24.449288+0000 mon.smithi163 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/trash_purge_schedule"}]: dispatch 2023-11-15T04:55:25.714 INFO:teuthology.orchestra.run.smithi163.stderr:mgr epoch 12 is available 2023-11-15T04:55:25.714 INFO:teuthology.orchestra.run.smithi163.stderr:Generating a dashboard self-signed certificate... 2023-11-15T04:55:25.714 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2023-11-15T04:55:26.211 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.208+0000 7fc3932b9700 1 Processor -- start 2023-11-15T04:55:26.211 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.208+0000 7fc3932b9700 1 -- start start 2023-11-15T04:55:26.212 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.208+0000 7fc3932b9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc38c0703d0 0x7fc38c100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:26.212 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.208+0000 7fc3932b9700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c0fc730 con 0x7fc38c0703d0 2023-11-15T04:55:26.212 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.209+0000 7fc391055700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc38c0703d0 0x7fc38c100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:26.212 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.209+0000 7fc391055700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc38c0703d0 0x7fc38c100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43340/0 (socket says 172.21.15.163:43340) 2023-11-15T04:55:26.212 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.209+0000 7fc391055700 1 -- 172.21.15.163:0/2837857016 learned_addr learned my addr 172.21.15.163:0/2837857016 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:26.213 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.209+0000 7fc391055700 1 -- 172.21.15.163:0/2837857016 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38c101230 con 0x7fc38c0703d0 2023-11-15T04:55:26.213 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.209+0000 7fc391055700 1 --2- 172.21.15.163:0/2837857016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc38c0703d0 0x7fc38c100d40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fc37c00b5f0 tx=0x7fc37c006ee0).ready entity=mon.0 client_cookie=691a531fda26d9a8 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:26.213 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.209+0000 7fc37bfff700 1 -- 172.21.15.163:0/2837857016 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc37c00d600 con 0x7fc38c0703d0 2023-11-15T04:55:26.213 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.209+0000 7fc37bfff700 1 -- 172.21.15.163:0/2837857016 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc37c00dc10 con 0x7fc38c0703d0 2023-11-15T04:55:26.213 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.209+0000 7fc37bfff700 1 -- 172.21.15.163:0/2837857016 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc37c011cd0 con 0x7fc38c0703d0 2023-11-15T04:55:26.215 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.210+0000 7fc3932b9700 1 -- 172.21.15.163:0/2837857016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc38c0703d0 msgr2=0x7fc38c100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:26.216 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.210+0000 7fc3932b9700 1 --2- 172.21.15.163:0/2837857016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc38c0703d0 0x7fc38c100d40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fc37c00b5f0 tx=0x7fc37c006ee0).stop 2023-11-15T04:55:26.216 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.210+0000 7fc3932b9700 1 -- 172.21.15.163:0/2837857016 shutdown_connections 2023-11-15T04:55:26.216 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.210+0000 7fc3932b9700 1 --2- 172.21.15.163:0/2837857016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc38c0703d0 0x7fc38c100d40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:26.216 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.210+0000 7fc3932b9700 1 -- 172.21.15.163:0/2837857016 >> 172.21.15.163:0/2837857016 conn(0x7fc38c0f63c0 msgr2=0x7fc38c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:26.216 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.210+0000 7fc3932b9700 1 -- 172.21.15.163:0/2837857016 shutdown_connections 2023-11-15T04:55:26.217 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.210+0000 7fc3932b9700 1 -- 172.21.15.163:0/2837857016 wait complete. 2023-11-15T04:55:26.217 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.211+0000 7fc3932b9700 1 Processor -- start 2023-11-15T04:55:26.217 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.211+0000 7fc3932b9700 1 -- start start 2023-11-15T04:55:26.217 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.211+0000 7fc3932b9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc38c0703d0 0x7fc38c10cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:26.217 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.211+0000 7fc3932b9700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c10d1f0 con 0x7fc38c0703d0 2023-11-15T04:55:26.218 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.211+0000 7fc391055700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc38c0703d0 0x7fc38c10cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:26.218 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.211+0000 7fc391055700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc38c0703d0 0x7fc38c10cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43352/0 (socket says 172.21.15.163:43352) 2023-11-15T04:55:26.218 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.211+0000 7fc391055700 1 -- 172.21.15.163:0/2925006612 learned_addr learned my addr 172.21.15.163:0/2925006612 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:26.218 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.212+0000 7fc391055700 1 -- 172.21.15.163:0/2925006612 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37c00b200 con 0x7fc38c0703d0 2023-11-15T04:55:26.218 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.212+0000 7fc391055700 1 --2- 172.21.15.163:0/2925006612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc38c0703d0 0x7fc38c10cd00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fc37c006a40 tx=0x7fc37c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:26.219 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.212+0000 7fc379ffb700 1 -- 172.21.15.163:0/2925006612 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc37c011c90 con 0x7fc38c0703d0 2023-11-15T04:55:26.219 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.212+0000 7fc3932b9700 1 -- 172.21.15.163:0/2925006612 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc38c10d3f0 con 0x7fc38c0703d0 2023-11-15T04:55:26.219 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.212+0000 7fc379ffb700 1 -- 172.21.15.163:0/2925006612 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc37c01b8c0 con 0x7fc38c0703d0 2023-11-15T04:55:26.219 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.212+0000 7fc379ffb700 1 -- 172.21.15.163:0/2925006612 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc37c01a8c0 con 0x7fc38c0703d0 2023-11-15T04:55:26.219 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.212+0000 7fc3932b9700 1 -- 172.21.15.163:0/2925006612 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc38c10d800 con 0x7fc38c0703d0 2023-11-15T04:55:26.220 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.213+0000 7fc379ffb700 1 -- 172.21.15.163:0/2925006612 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fc37c00f070 con 0x7fc38c0703d0 2023-11-15T04:55:26.220 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.213+0000 7fc379ffb700 1 --2- 172.21.15.163:0/2925006612 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc37402fd70 0x7fc374032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:26.220 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.213+0000 7fc379ffb700 1 -- 172.21.15.163:0/2925006612 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc37c040a10 con 0x7fc38c0703d0 2023-11-15T04:55:26.220 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.213+0000 7fc3932b9700 1 -- 172.21.15.163:0/2925006612 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc38c045ad0 con 0x7fc38c0703d0 2023-11-15T04:55:26.220 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.213+0000 7fc390854700 1 --2- 172.21.15.163:0/2925006612 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc37402fd70 0x7fc374032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:26.221 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.213+0000 7fc390854700 1 --2- 172.21.15.163:0/2925006612 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc37402fd70 0x7fc374032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fc3880096a0 tx=0x7fc388006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:26.221 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.218+0000 7fc379ffb700 1 -- 172.21.15.163:0/2925006612 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc37c016070 con 0x7fc38c0703d0 2023-11-15T04:55:26.323 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.322+0000 7fc379ffb700 1 -- 172.21.15.163:0/2925006612 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fc37c024550 con 0x7fc38c0703d0 2023-11-15T04:55:26.382 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.381+0000 7fc3932b9700 1 -- 172.21.15.163:0/2925006612 --> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fc38c10ed90 con 0x7fc37402fd70 2023-11-15T04:55:26.504 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.502+0000 7fc379ffb700 1 -- 172.21.15.163:0/2925006612 <== mgr.14162 v2:172.21.15.163:6800/3399151648 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fc38c10ed90 con 0x7fc37402fd70 2023-11-15T04:55:26.504 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: Self-signed certificate created 2023-11-15T04:55:26.506 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.504+0000 7fc3932b9700 1 -- 172.21.15.163:0/2925006612 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc37402fd70 msgr2=0x7fc374032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:26.506 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.504+0000 7fc3932b9700 1 --2- 172.21.15.163:0/2925006612 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc37402fd70 0x7fc374032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fc3880096a0 tx=0x7fc388006b40).stop 2023-11-15T04:55:26.506 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.504+0000 7fc3932b9700 1 -- 172.21.15.163:0/2925006612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc38c0703d0 msgr2=0x7fc38c10cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:26.507 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.504+0000 7fc3932b9700 1 --2- 172.21.15.163:0/2925006612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc38c0703d0 0x7fc38c10cd00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fc37c006a40 tx=0x7fc37c006ee0).stop 2023-11-15T04:55:26.507 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.504+0000 7fc3932b9700 1 -- 172.21.15.163:0/2925006612 shutdown_connections 2023-11-15T04:55:26.507 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.504+0000 7fc3932b9700 1 --2- 172.21.15.163:0/2925006612 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc37402fd70 0x7fc374032230 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:26.507 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.504+0000 7fc3932b9700 1 --2- 172.21.15.163:0/2925006612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc38c0703d0 0x7fc38c10cd00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:26.507 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.504+0000 7fc3932b9700 1 -- 172.21.15.163:0/2925006612 >> 172.21.15.163:0/2925006612 conn(0x7fc38c0f63c0 msgr2=0x7fc38c0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:26.508 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.504+0000 7fc3932b9700 1 -- 172.21.15.163:0/2925006612 shutdown_connections 2023-11-15T04:55:26.508 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:26.504+0000 7fc3932b9700 1 -- 172.21.15.163:0/2925006612 wait complete. 2023-11-15T04:55:26.670 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:26 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:25.313279+0000 mgr.smithi163.qesyrx (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-15T04:55:26.671 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:26 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:25.313819+0000 mon.smithi163 (mon.0) 105 : cluster [DBG] mgrmap e14: smithi163.qesyrx(active, since 1.06034s) 2023-11-15T04:55:26.671 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:26 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:25.320814+0000 mgr.smithi163.qesyrx (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-15T04:55:26.671 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:26 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:25.323447+0000 mon.smithi163 (mon.0) 106 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' 2023-11-15T04:55:26.838 INFO:teuthology.orchestra.run.smithi163.stderr:Creating initial admin user... 2023-11-15T04:55:26.839 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp5ozx282v:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2023-11-15T04:55:27.163 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.159+0000 7f63c9628700 1 Processor -- start 2023-11-15T04:55:27.164 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.159+0000 7f63c9628700 1 -- start start 2023-11-15T04:55:27.164 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.159+0000 7f63c9628700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63c4072500 0x7f63c4070aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:27.164 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.159+0000 7f63c9628700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4072920 con 0x7f63c4072500 2023-11-15T04:55:27.165 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.160+0000 7f63c2ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63c4072500 0x7f63c4070aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:27.165 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.160+0000 7f63c2ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63c4072500 0x7f63c4070aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43356/0 (socket says 172.21.15.163:43356) 2023-11-15T04:55:27.165 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.160+0000 7f63c2ffd700 1 -- 172.21.15.163:0/2582711049 learned_addr learned my addr 172.21.15.163:0/2582711049 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:27.165 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.160+0000 7f63c2ffd700 1 -- 172.21.15.163:0/2582711049 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c4070f90 con 0x7f63c4072500 2023-11-15T04:55:27.165 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.160+0000 7f63c2ffd700 1 --2- 172.21.15.163:0/2582711049 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63c4072500 0x7f63c4070aa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f63b800b5f0 tx=0x7f63b8006ee0).ready entity=mon.0 client_cookie=af4180f1aded9b44 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:27.166 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.160+0000 7f63c1ffb700 1 -- 172.21.15.163:0/2582711049 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63b800d600 con 0x7f63c4072500 2023-11-15T04:55:27.166 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.160+0000 7f63c1ffb700 1 -- 172.21.15.163:0/2582711049 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63b800dc10 con 0x7f63c4072500 2023-11-15T04:55:27.166 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.160+0000 7f63c1ffb700 1 -- 172.21.15.163:0/2582711049 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63b8011cd0 con 0x7f63c4072500 2023-11-15T04:55:27.166 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.161+0000 7f63c9628700 1 -- 172.21.15.163:0/2582711049 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63c4072500 msgr2=0x7f63c4070aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:27.166 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.161+0000 7f63c9628700 1 --2- 172.21.15.163:0/2582711049 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63c4072500 0x7f63c4070aa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f63b800b5f0 tx=0x7f63b8006ee0).stop 2023-11-15T04:55:27.167 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.161+0000 7f63c9628700 1 -- 172.21.15.163:0/2582711049 shutdown_connections 2023-11-15T04:55:27.167 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.161+0000 7f63c9628700 1 --2- 172.21.15.163:0/2582711049 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63c4072500 0x7f63c4070aa0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:27.167 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.161+0000 7f63c9628700 1 -- 172.21.15.163:0/2582711049 >> 172.21.15.163:0/2582711049 conn(0x7f63c40f74b0 msgr2=0x7f63c40f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:27.167 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.161+0000 7f63c9628700 1 -- 172.21.15.163:0/2582711049 shutdown_connections 2023-11-15T04:55:27.168 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.161+0000 7f63c9628700 1 -- 172.21.15.163:0/2582711049 wait complete. 2023-11-15T04:55:27.168 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.162+0000 7f63c9628700 1 Processor -- start 2023-11-15T04:55:27.168 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.162+0000 7f63c9628700 1 -- start start 2023-11-15T04:55:27.168 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.162+0000 7f63c9628700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63c4072500 0x7f63c410cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:27.168 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.162+0000 7f63c9628700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c410d230 con 0x7f63c4072500 2023-11-15T04:55:27.169 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.162+0000 7f63c2ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63c4072500 0x7f63c410cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:27.169 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.162+0000 7f63c2ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63c4072500 0x7f63c410cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43364/0 (socket says 172.21.15.163:43364) 2023-11-15T04:55:27.169 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.162+0000 7f63c2ffd700 1 -- 172.21.15.163:0/3780138223 learned_addr learned my addr 172.21.15.163:0/3780138223 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:27.169 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.162+0000 7f63c2ffd700 1 -- 172.21.15.163:0/3780138223 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b800b200 con 0x7f63c4072500 2023-11-15T04:55:27.170 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.162+0000 7f63c2ffd700 1 --2- 172.21.15.163:0/3780138223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63c4072500 0x7f63c410cd40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f63b8006a40 tx=0x7f63b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:27.171 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.163+0000 7f63ab7fe700 1 -- 172.21.15.163:0/3780138223 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63b8011c90 con 0x7f63c4072500 2023-11-15T04:55:27.172 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.163+0000 7f63c9628700 1 -- 172.21.15.163:0/3780138223 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c410d430 con 0x7f63c4072500 2023-11-15T04:55:27.172 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.163+0000 7f63ab7fe700 1 -- 172.21.15.163:0/3780138223 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63b801b8c0 con 0x7f63c4072500 2023-11-15T04:55:27.172 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.163+0000 7f63ab7fe700 1 -- 172.21.15.163:0/3780138223 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63b801a8c0 con 0x7f63c4072500 2023-11-15T04:55:27.172 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.163+0000 7f63c9628700 1 -- 172.21.15.163:0/3780138223 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c410d840 con 0x7f63c4072500 2023-11-15T04:55:27.172 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.163+0000 7f63ab7fe700 1 -- 172.21.15.163:0/3780138223 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f63b800f070 con 0x7f63c4072500 2023-11-15T04:55:27.173 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.164+0000 7f63ab7fe700 1 --2- 172.21.15.163:0/3780138223 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f63b002feb0 0x7f63b0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:27.173 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.164+0000 7f63ab7fe700 1 -- 172.21.15.163:0/3780138223 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f63b8040c80 con 0x7f63c4072500 2023-11-15T04:55:27.173 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.164+0000 7f63c27fc700 1 --2- 172.21.15.163:0/3780138223 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f63b002feb0 0x7f63b0032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:27.173 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.164+0000 7f63c9628700 1 -- 172.21.15.163:0/3780138223 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c4101e60 con 0x7f63c4072500 2023-11-15T04:55:27.173 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.164+0000 7f63c27fc700 1 --2- 172.21.15.163:0/3780138223 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f63b002feb0 0x7f63b0032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f63b40096a0 tx=0x7f63b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:27.174 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.168+0000 7f63ab7fe700 1 -- 172.21.15.163:0/3780138223 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63b8016070 con 0x7f63c4072500 2023-11-15T04:55:27.337 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.336+0000 7f63c9628700 1 -- 172.21.15.163:0/3780138223 --> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f63c4057f10 con 0x7f63b002feb0 2023-11-15T04:55:27.554 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.553+0000 7f63ab7fe700 1 -- 172.21.15.163:0/3780138223 <== mgr.14162 v2:172.21.15.163:6800/3399151648 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f63c4057f10 con 0x7f63b002feb0 2023-11-15T04:55:27.555 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$VuBSL6wI6LqYzzGJNw005O/v.H4e9ATsuhsvo1Tw2Vh4tNdqHrrsW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1700024127, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-11-15T04:55:27.556 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.554+0000 7f63c9628700 1 -- 172.21.15.163:0/3780138223 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f63b002feb0 msgr2=0x7f63b0032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:27.557 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.554+0000 7f63c9628700 1 --2- 172.21.15.163:0/3780138223 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f63b002feb0 0x7f63b0032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f63b40096a0 tx=0x7f63b4006b40).stop 2023-11-15T04:55:27.557 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.554+0000 7f63c9628700 1 -- 172.21.15.163:0/3780138223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63c4072500 msgr2=0x7f63c410cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:27.557 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.554+0000 7f63c9628700 1 --2- 172.21.15.163:0/3780138223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63c4072500 0x7f63c410cd40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f63b8006a40 tx=0x7f63b8006ee0).stop 2023-11-15T04:55:27.557 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.555+0000 7f63c9628700 1 -- 172.21.15.163:0/3780138223 shutdown_connections 2023-11-15T04:55:27.558 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.555+0000 7f63c9628700 1 --2- 172.21.15.163:0/3780138223 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f63b002feb0 0x7f63b0032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:27.558 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.555+0000 7f63c9628700 1 --2- 172.21.15.163:0/3780138223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63c4072500 0x7f63c410cd40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:27.558 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.555+0000 7f63c9628700 1 -- 172.21.15.163:0/3780138223 >> 172.21.15.163:0/3780138223 conn(0x7f63c40f74b0 msgr2=0x7f63c40f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:27.558 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.555+0000 7f63c9628700 1 -- 172.21.15.163:0/3780138223 shutdown_connections 2023-11-15T04:55:27.559 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:27.555+0000 7f63c9628700 1 -- 172.21.15.163:0/3780138223 wait complete. 2023-11-15T04:55:27.584 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:27 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:26.324379+0000 mon.smithi163 (mon.0) 107 : cluster [DBG] mgrmap e15: smithi163.qesyrx(active, since 2s) 2023-11-15T04:55:27.584 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:27 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:26.383020+0000 mgr.smithi163.qesyrx (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:55:27.585 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:27 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:26.501911+0000 mon.smithi163 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' 2023-11-15T04:55:27.585 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:27 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:26.503746+0000 mon.smithi163 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' 2023-11-15T04:55:27.913 INFO:teuthology.orchestra.run.smithi163.stderr:Fetching dashboard port number... 2023-11-15T04:55:27.913 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpdxbghfxu:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5hx8597a:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2023-11-15T04:55:28.289 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.285+0000 7f149f73c700 1 Processor -- start 2023-11-15T04:55:28.289 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.285+0000 7f149f73c700 1 -- start start 2023-11-15T04:55:28.289 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.285+0000 7f149f73c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14980fab80 0x7f14980fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:28.289 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.285+0000 7f149f73c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14980fb490 con 0x7f14980fab80 2023-11-15T04:55:28.290 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.286+0000 7f149d4d8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14980fab80 0x7f14980fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:28.290 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.286+0000 7f149d4d8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14980fab80 0x7f14980fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43368/0 (socket says 172.21.15.163:43368) 2023-11-15T04:55:28.290 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.286+0000 7f149d4d8700 1 -- 172.21.15.163:0/1436743538 learned_addr learned my addr 172.21.15.163:0/1436743538 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:28.290 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.286+0000 7f149d4d8700 1 -- 172.21.15.163:0/1436743538 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14980fb5d0 con 0x7f14980fab80 2023-11-15T04:55:28.290 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.286+0000 7f149d4d8700 1 --2- 172.21.15.163:0/1436743538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14980fab80 0x7f14980fafa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f148800b5f0 tx=0x7f1488006ee0).ready entity=mon.0 client_cookie=746c9ed0bcac9b15 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:28.291 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.286+0000 7f1487fff700 1 -- 172.21.15.163:0/1436743538 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f148800d600 con 0x7f14980fab80 2023-11-15T04:55:28.291 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.286+0000 7f1487fff700 1 -- 172.21.15.163:0/1436743538 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f148800dc10 con 0x7f14980fab80 2023-11-15T04:55:28.291 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.286+0000 7f1487fff700 1 -- 172.21.15.163:0/1436743538 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1488011cd0 con 0x7f14980fab80 2023-11-15T04:55:28.291 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.287+0000 7f149f73c700 1 -- 172.21.15.163:0/1436743538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14980fab80 msgr2=0x7f14980fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:28.291 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.287+0000 7f149f73c700 1 --2- 172.21.15.163:0/1436743538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14980fab80 0x7f14980fafa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f148800b5f0 tx=0x7f1488006ee0).stop 2023-11-15T04:55:28.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.287+0000 7f149f73c700 1 -- 172.21.15.163:0/1436743538 shutdown_connections 2023-11-15T04:55:28.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.287+0000 7f149f73c700 1 --2- 172.21.15.163:0/1436743538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14980fab80 0x7f14980fafa0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:28.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.287+0000 7f149f73c700 1 -- 172.21.15.163:0/1436743538 >> 172.21.15.163:0/1436743538 conn(0x7f14980f63c0 msgr2=0x7f14980f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:28.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.287+0000 7f149f73c700 1 -- 172.21.15.163:0/1436743538 shutdown_connections 2023-11-15T04:55:28.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.287+0000 7f149f73c700 1 -- 172.21.15.163:0/1436743538 wait complete. 2023-11-15T04:55:28.293 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.288+0000 7f149f73c700 1 Processor -- start 2023-11-15T04:55:28.293 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.288+0000 7f149f73c700 1 -- start start 2023-11-15T04:55:28.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.288+0000 7f149f73c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14980fab80 0x7f149810aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:28.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.288+0000 7f149f73c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149810af20 con 0x7f14980fab80 2023-11-15T04:55:28.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.288+0000 7f149d4d8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14980fab80 0x7f149810aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:28.296 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.288+0000 7f149d4d8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14980fab80 0x7f149810aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43374/0 (socket says 172.21.15.163:43374) 2023-11-15T04:55:28.296 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.288+0000 7f149d4d8700 1 -- 172.21.15.163:0/3453244193 learned_addr learned my addr 172.21.15.163:0/3453244193 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:28.296 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.288+0000 7f149d4d8700 1 -- 172.21.15.163:0/3453244193 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148800b200 con 0x7f14980fab80 2023-11-15T04:55:28.296 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.288+0000 7f149d4d8700 1 --2- 172.21.15.163:0/3453244193 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14980fab80 0x7f149810aa30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f148800e040 tx=0x7f1488006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:28.297 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.289+0000 7f1485ffb700 1 -- 172.21.15.163:0/3453244193 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f148801b410 con 0x7f14980fab80 2023-11-15T04:55:28.297 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.289+0000 7f1485ffb700 1 -- 172.21.15.163:0/3453244193 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f148801ba20 con 0x7f14980fab80 2023-11-15T04:55:28.297 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.289+0000 7f149f73c700 1 -- 172.21.15.163:0/3453244193 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f149810b120 con 0x7f14980fab80 2023-11-15T04:55:28.297 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.289+0000 7f1485ffb700 1 -- 172.21.15.163:0/3453244193 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f148801a910 con 0x7f14980fab80 2023-11-15T04:55:28.297 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.289+0000 7f149f73c700 1 -- 172.21.15.163:0/3453244193 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f149810b530 con 0x7f14980fab80 2023-11-15T04:55:28.298 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.290+0000 7f1485ffb700 1 -- 172.21.15.163:0/3453244193 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f148800f070 con 0x7f14980fab80 2023-11-15T04:55:28.298 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.290+0000 7f1485ffb700 1 --2- 172.21.15.163:0/3453244193 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f148002feb0 0x7f1480032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:28.298 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.290+0000 7f149f73c700 1 -- 172.21.15.163:0/3453244193 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1498045ad0 con 0x7f14980fab80 2023-11-15T04:55:28.298 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.290+0000 7f1485ffb700 1 -- 172.21.15.163:0/3453244193 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1488041320 con 0x7f14980fab80 2023-11-15T04:55:28.298 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.290+0000 7f149ccd7700 1 --2- 172.21.15.163:0/3453244193 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f148002feb0 0x7f1480032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:28.299 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.290+0000 7f149ccd7700 1 --2- 172.21.15.163:0/3453244193 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f148002feb0 0x7f1480032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f14940096a0 tx=0x7f1494006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:28.299 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.294+0000 7f1485ffb700 1 -- 172.21.15.163:0/3453244193 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1488016070 con 0x7f14980fab80 2023-11-15T04:55:28.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.445+0000 7f149f73c700 1 -- 172.21.15.163:0/3453244193 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f14980fc900 con 0x7f14980fab80 2023-11-15T04:55:28.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.446+0000 7f1485ffb700 1 -- 172.21.15.163:0/3453244193 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f1488010540 con 0x7f14980fab80 2023-11-15T04:55:28.448 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 8443 2023-11-15T04:55:28.450 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.448+0000 7f149f73c700 1 -- 172.21.15.163:0/3453244193 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f148002feb0 msgr2=0x7f1480032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:28.450 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.448+0000 7f149f73c700 1 --2- 172.21.15.163:0/3453244193 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f148002feb0 0x7f1480032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f14940096a0 tx=0x7f1494006b40).stop 2023-11-15T04:55:28.450 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.448+0000 7f149f73c700 1 -- 172.21.15.163:0/3453244193 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14980fab80 msgr2=0x7f149810aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:28.451 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.448+0000 7f149f73c700 1 --2- 172.21.15.163:0/3453244193 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14980fab80 0x7f149810aa30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f148800e040 tx=0x7f1488006ee0).stop 2023-11-15T04:55:28.451 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.448+0000 7f149f73c700 1 -- 172.21.15.163:0/3453244193 shutdown_connections 2023-11-15T04:55:28.451 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.448+0000 7f149f73c700 1 --2- 172.21.15.163:0/3453244193 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f148002feb0 0x7f1480032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:28.451 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.448+0000 7f149f73c700 1 --2- 172.21.15.163:0/3453244193 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14980fab80 0x7f149810aa30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:28.451 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.448+0000 7f149f73c700 1 -- 172.21.15.163:0/3453244193 >> 172.21.15.163:0/3453244193 conn(0x7f14980f63c0 msgr2=0x7f14980f6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:28.451 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.448+0000 7f149f73c700 1 -- 172.21.15.163:0/3453244193 shutdown_connections 2023-11-15T04:55:28.452 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2023-11-15T04:55:28.448+0000 7f149f73c700 1 -- 172.21.15.163:0/3453244193 wait complete. 2023-11-15T04:55:28.814 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T04:55:28.820 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: disabled 2023-11-15T04:55:28.821 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T04:55:28.828 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: inactive 2023-11-15T04:55:28.828 INFO:teuthology.orchestra.run.smithi163.stderr:firewalld.service is not enabled 2023-11-15T04:55:28.829 INFO:teuthology.orchestra.run.smithi163.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-11-15T04:55:28.831 INFO:teuthology.orchestra.run.smithi163.stderr:Ceph Dashboard is now available at: 2023-11-15T04:55:28.831 INFO:teuthology.orchestra.run.smithi163.stderr: 2023-11-15T04:55:28.831 INFO:teuthology.orchestra.run.smithi163.stderr: URL: https://smithi163.front.sepia.ceph.com:8443/ 2023-11-15T04:55:28.832 INFO:teuthology.orchestra.run.smithi163.stderr: User: admin 2023-11-15T04:55:28.832 INFO:teuthology.orchestra.run.smithi163.stderr: Password: 5x8tmbjaz4 2023-11-15T04:55:28.832 INFO:teuthology.orchestra.run.smithi163.stderr: 2023-11-15T04:55:28.833 INFO:teuthology.orchestra.run.smithi163.stderr:You can access the Ceph CLI with: 2023-11-15T04:55:28.834 INFO:teuthology.orchestra.run.smithi163.stderr: 2023-11-15T04:55:28.834 INFO:teuthology.orchestra.run.smithi163.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-11-15T04:55:28.834 INFO:teuthology.orchestra.run.smithi163.stderr: 2023-11-15T04:55:28.834 INFO:teuthology.orchestra.run.smithi163.stderr:Please consider enabling telemetry to help improve Ceph: 2023-11-15T04:55:28.835 INFO:teuthology.orchestra.run.smithi163.stderr: 2023-11-15T04:55:28.835 INFO:teuthology.orchestra.run.smithi163.stderr: ceph telemetry on 2023-11-15T04:55:28.835 INFO:teuthology.orchestra.run.smithi163.stderr: 2023-11-15T04:55:28.835 INFO:teuthology.orchestra.run.smithi163.stderr:For more information see: 2023-11-15T04:55:28.835 INFO:teuthology.orchestra.run.smithi163.stderr: 2023-11-15T04:55:28.836 INFO:teuthology.orchestra.run.smithi163.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2023-11-15T04:55:28.836 INFO:teuthology.orchestra.run.smithi163.stderr: 2023-11-15T04:55:28.836 INFO:teuthology.orchestra.run.smithi163.stderr:Bootstrap complete. 2023-11-15T04:55:28.837 INFO:teuthology.orchestra.run.smithi163.stderr:Releasing lock 140099707816424 on /run/cephadm/01689f7e-8373-11ee-8dbe-212e2dc638e7.lock 2023-11-15T04:55:28.837 INFO:teuthology.orchestra.run.smithi163.stderr:Lock 140099707816424 released on /run/cephadm/01689f7e-8373-11ee-8dbe-212e2dc638e7.lock 2023-11-15T04:55:28.851 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:28 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:27.338160+0000 mgr.smithi163.qesyrx (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:55:28.851 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:28 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:27.554415+0000 mon.smithi163 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' 2023-11-15T04:55:28.851 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:28 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:28.447575+0000 mon.smithi163 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.163:0/3453244193' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-11-15T04:55:28.865 INFO:tasks.cephadm:Fetching config... 2023-11-15T04:55:28.866 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-11-15T04:55:28.866 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-11-15T04:55:28.886 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-11-15T04:55:28.887 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-11-15T04:55:28.887 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-11-15T04:55:28.945 INFO:tasks.cephadm:Fetching mon keyring... 2023-11-15T04:55:28.946 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-11-15T04:55:28.946 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/mon.smithi163/keyring of=/dev/stdout 2023-11-15T04:55:29.013 INFO:tasks.cephadm:Fetching pub ssh key... 2023-11-15T04:55:29.013 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-11-15T04:55:29.014 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-11-15T04:55:29.072 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-11-15T04:55:29.072 DEBUG:teuthology.orchestra.run.smithi163:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCh0oJUIrrpSBox3sxjwxB0vqh1Jf8sMaw/tuitAfQKRUHPsNamy3MCuZ5W1n7nDoBNSPQB+qd1hL1AkZJg3dnHL6OHwwIWTkKWHOBZdYIOZhu9XJtdEWb2aHQ5Pjm5ikby+r7zLBvtnmvpCn06z1BDVIa6sozeajmkVjcWlZZZ1JVeasmNdRA30CZFTicm16qxGwPZYycFmGmgL1bxYRTptOZISQygCZKPoDzIJrON/tw6sQVbuh/vp0fArlGbGoW2enqXCDb9nvSIjSqU1L2EbA1Kp5bn0i3yn5BrQ6qe+C16MPTT3YFngYtazjExYCdefardidOHHhzl0EkuobA9DkAI017WA0kvVVfpAhMc/1rh86zdptf5pb5SEEz8pQoYLhgysliqs5GhSu+0iGR1aH3N2ifE3ryybGFx+15ObJVx58sozwjVTzWMs2sCeMywQ2jpRxux2YUQ1VH0qv3WFQt9PD3ob/Xdm25FDh6nNf4kLP3EWG9+guGCpy4aKs= ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-15T04:55:29.149 INFO:teuthology.orchestra.run.smithi163.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCh0oJUIrrpSBox3sxjwxB0vqh1Jf8sMaw/tuitAfQKRUHPsNamy3MCuZ5W1n7nDoBNSPQB+qd1hL1AkZJg3dnHL6OHwwIWTkKWHOBZdYIOZhu9XJtdEWb2aHQ5Pjm5ikby+r7zLBvtnmvpCn06z1BDVIa6sozeajmkVjcWlZZZ1JVeasmNdRA30CZFTicm16qxGwPZYycFmGmgL1bxYRTptOZISQygCZKPoDzIJrON/tw6sQVbuh/vp0fArlGbGoW2enqXCDb9nvSIjSqU1L2EbA1Kp5bn0i3yn5BrQ6qe+C16MPTT3YFngYtazjExYCdefardidOHHhzl0EkuobA9DkAI017WA0kvVVfpAhMc/1rh86zdptf5pb5SEEz8pQoYLhgysliqs5GhSu+0iGR1aH3N2ifE3ryybGFx+15ObJVx58sozwjVTzWMs2sCeMywQ2jpRxux2YUQ1VH0qv3WFQt9PD3ob/Xdm25FDh6nNf4kLP3EWG9+guGCpy4aKs= ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7 2023-11-15T04:55:29.160 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCh0oJUIrrpSBox3sxjwxB0vqh1Jf8sMaw/tuitAfQKRUHPsNamy3MCuZ5W1n7nDoBNSPQB+qd1hL1AkZJg3dnHL6OHwwIWTkKWHOBZdYIOZhu9XJtdEWb2aHQ5Pjm5ikby+r7zLBvtnmvpCn06z1BDVIa6sozeajmkVjcWlZZZ1JVeasmNdRA30CZFTicm16qxGwPZYycFmGmgL1bxYRTptOZISQygCZKPoDzIJrON/tw6sQVbuh/vp0fArlGbGoW2enqXCDb9nvSIjSqU1L2EbA1Kp5bn0i3yn5BrQ6qe+C16MPTT3YFngYtazjExYCdefardidOHHhzl0EkuobA9DkAI017WA0kvVVfpAhMc/1rh86zdptf5pb5SEEz8pQoYLhgysliqs5GhSu+0iGR1aH3N2ifE3ryybGFx+15ObJVx58sozwjVTzWMs2sCeMywQ2jpRxux2YUQ1VH0qv3WFQt9PD3ob/Xdm25FDh6nNf4kLP3EWG9+guGCpy4aKs= ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-15T04:55:29.206 INFO:teuthology.orchestra.run.smithi178.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCh0oJUIrrpSBox3sxjwxB0vqh1Jf8sMaw/tuitAfQKRUHPsNamy3MCuZ5W1n7nDoBNSPQB+qd1hL1AkZJg3dnHL6OHwwIWTkKWHOBZdYIOZhu9XJtdEWb2aHQ5Pjm5ikby+r7zLBvtnmvpCn06z1BDVIa6sozeajmkVjcWlZZZ1JVeasmNdRA30CZFTicm16qxGwPZYycFmGmgL1bxYRTptOZISQygCZKPoDzIJrON/tw6sQVbuh/vp0fArlGbGoW2enqXCDb9nvSIjSqU1L2EbA1Kp5bn0i3yn5BrQ6qe+C16MPTT3YFngYtazjExYCdefardidOHHhzl0EkuobA9DkAI017WA0kvVVfpAhMc/1rh86zdptf5pb5SEEz8pQoYLhgysliqs5GhSu+0iGR1aH3N2ifE3ryybGFx+15ObJVx58sozwjVTzWMs2sCeMywQ2jpRxux2YUQ1VH0qv3WFQt9PD3ob/Xdm25FDh6nNf4kLP3EWG9+guGCpy4aKs= ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7 2023-11-15T04:55:29.218 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-11-15T04:55:31.343 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.341+0000 7f30f2268700 1 -- 172.21.15.163:0/3437554156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30ec0f51a0 msgr2=0x7f30ec0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:31.343 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.341+0000 7f30f2268700 1 --2- 172.21.15.163:0/3437554156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30ec0f51a0 0x7f30ec0f55c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f30e8009fb0 tx=0x7f30e8009b30).stop 2023-11-15T04:55:31.343 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.341+0000 7f30f2268700 1 -- 172.21.15.163:0/3437554156 shutdown_connections 2023-11-15T04:55:31.344 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.341+0000 7f30f2268700 1 --2- 172.21.15.163:0/3437554156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30ec0f51a0 0x7f30ec0f55c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:31.344 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.341+0000 7f30f2268700 1 -- 172.21.15.163:0/3437554156 >> 172.21.15.163:0/3437554156 conn(0x7f30ec0f09a0 msgr2=0x7f30ec0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:31.344 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.341+0000 7f30f2268700 1 -- 172.21.15.163:0/3437554156 shutdown_connections 2023-11-15T04:55:31.344 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.341+0000 7f30f2268700 1 -- 172.21.15.163:0/3437554156 wait complete. 2023-11-15T04:55:31.344 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.342+0000 7f30f2268700 1 Processor -- start 2023-11-15T04:55:31.345 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.342+0000 7f30f2268700 1 -- start start 2023-11-15T04:55:31.345 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.342+0000 7f30f2268700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30ec0f51a0 0x7f30ec1064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:31.345 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.342+0000 7f30f2268700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec106990 con 0x7f30ec0f51a0 2023-11-15T04:55:31.345 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.342+0000 7f30f1266700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30ec0f51a0 0x7f30ec1064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:31.345 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.342+0000 7f30f1266700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30ec0f51a0 0x7f30ec1064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34524/0 (socket says 172.21.15.163:34524) 2023-11-15T04:55:31.346 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.342+0000 7f30f1266700 1 -- 172.21.15.163:0/1005110380 learned_addr learned my addr 172.21.15.163:0/1005110380 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:31.346 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.342+0000 7f30f1266700 1 -- 172.21.15.163:0/1005110380 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e8005040 con 0x7f30ec0f51a0 2023-11-15T04:55:31.346 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.342+0000 7f30f1266700 1 --2- 172.21.15.163:0/1005110380 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30ec0f51a0 0x7f30ec1064a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f30e8009830 tx=0x7f30e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:31.346 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.342+0000 7f30e1ffb700 1 -- 172.21.15.163:0/1005110380 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30e8009320 con 0x7f30ec0f51a0 2023-11-15T04:55:31.346 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.342+0000 7f30e1ffb700 1 -- 172.21.15.163:0/1005110380 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30e80136c0 con 0x7f30ec0f51a0 2023-11-15T04:55:31.347 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.342+0000 7f30f2268700 1 -- 172.21.15.163:0/1005110380 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ec106b90 con 0x7f30ec0f51a0 2023-11-15T04:55:31.347 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.342+0000 7f30e1ffb700 1 -- 172.21.15.163:0/1005110380 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30e801ee90 con 0x7f30ec0f51a0 2023-11-15T04:55:31.347 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.343+0000 7f30f2268700 1 -- 172.21.15.163:0/1005110380 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ec107020 con 0x7f30ec0f51a0 2023-11-15T04:55:31.348 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.343+0000 7f30e1ffb700 1 -- 172.21.15.163:0/1005110380 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30e8014420 con 0x7f30ec0f51a0 2023-11-15T04:55:31.348 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.343+0000 7f30e1ffb700 1 --2- 172.21.15.163:0/1005110380 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f30d802ff10 0x7f30d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:31.348 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.343+0000 7f30f2268700 1 -- 172.21.15.163:0/1005110380 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30ec045ad0 con 0x7f30ec0f51a0 2023-11-15T04:55:31.349 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.343+0000 7f30e1ffb700 1 -- 172.21.15.163:0/1005110380 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30e8040ed0 con 0x7f30ec0f51a0 2023-11-15T04:55:31.349 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.345+0000 7f30f0a65700 1 --2- 172.21.15.163:0/1005110380 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f30d802ff10 0x7f30d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:31.349 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.345+0000 7f30f0a65700 1 --2- 172.21.15.163:0/1005110380 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f30d802ff10 0x7f30d80323d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f30dc0096a0 tx=0x7f30dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:31.349 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.348+0000 7f30e1ffb700 1 -- 172.21.15.163:0/1005110380 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30e801c910 con 0x7f30ec0f51a0 2023-11-15T04:55:31.521 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.521+0000 7f30f2268700 1 -- 172.21.15.163:0/1005110380 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f30ec107ee0 con 0x7f30ec0f51a0 2023-11-15T04:55:31.524 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.524+0000 7f30e1ffb700 1 -- 172.21.15.163:0/1005110380 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f30e801c320 con 0x7f30ec0f51a0 2023-11-15T04:55:31.526 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.526+0000 7f30f2268700 1 -- 172.21.15.163:0/1005110380 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f30d802ff10 msgr2=0x7f30d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:31.526 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.526+0000 7f30f2268700 1 --2- 172.21.15.163:0/1005110380 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f30d802ff10 0x7f30d80323d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f30dc0096a0 tx=0x7f30dc006b40).stop 2023-11-15T04:55:31.527 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.526+0000 7f30f2268700 1 -- 172.21.15.163:0/1005110380 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30ec0f51a0 msgr2=0x7f30ec1064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:31.527 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.526+0000 7f30f2268700 1 --2- 172.21.15.163:0/1005110380 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30ec0f51a0 0x7f30ec1064a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f30e8009830 tx=0x7f30e800b040).stop 2023-11-15T04:55:31.527 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.526+0000 7f30f2268700 1 -- 172.21.15.163:0/1005110380 shutdown_connections 2023-11-15T04:55:31.527 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.526+0000 7f30f2268700 1 --2- 172.21.15.163:0/1005110380 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f30d802ff10 0x7f30d80323d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:31.528 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.526+0000 7f30f2268700 1 --2- 172.21.15.163:0/1005110380 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30ec0f51a0 0x7f30ec1064a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:31.528 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.526+0000 7f30f2268700 1 -- 172.21.15.163:0/1005110380 >> 172.21.15.163:0/1005110380 conn(0x7f30ec0f09a0 msgr2=0x7f30ec05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:31.528 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.526+0000 7f30f2268700 1 -- 172.21.15.163:0/1005110380 shutdown_connections 2023-11-15T04:55:31.528 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:31.526+0000 7f30f2268700 1 -- 172.21.15.163:0/1005110380 wait complete. 2023-11-15T04:55:31.603 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:31 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:30.331375+0000 mon.smithi163 (mon.0) 112 : cluster [DBG] mgrmap e16: smithi163.qesyrx(active, since 6s) 2023-11-15T04:55:31.941 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi178 2023-11-15T04:55:31.941 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-15T04:55:31.942 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/etc/ceph/ceph.conf 2023-11-15T04:55:31.964 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-11-15T04:55:31.964 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-15T04:55:32.026 INFO:tasks.cephadm:Adding host smithi178 to orchestrator... 2023-11-15T04:55:32.026 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph orch host add smithi178 2023-11-15T04:55:32.764 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:32 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:31.524770+0000 mon.smithi163 (mon.0) 113 : audit [INF] from='client.? 172.21.15.163:0/1005110380' entity='client.admin' 2023-11-15T04:55:34.114 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:34 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:33.034195+0000 mon.smithi163 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' 2023-11-15T04:55:34.115 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:34 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:33.311175+0000 mon.smithi163 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' 2023-11-15T04:55:35.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cephadm 2023-11-15T04:55:33.317623+0000 mgr.smithi163.qesyrx (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi163 on smithi163 2023-11-15T04:55:45.703 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:45 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:44.310637+0000 mgr.smithi163.qesyrx (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:55:48.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:47 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:46.311009+0000 mgr.smithi163.qesyrx (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:55:50.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:49 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:48.311350+0000 mgr.smithi163.qesyrx (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:55:52.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:51 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:50.311596+0000 mgr.smithi163.qesyrx (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:55:54.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:53 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:52.311792+0000 mgr.smithi163.qesyrx (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:55:54.873 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.871+0000 7fed1c4cd700 1 -- 172.21.15.178:0/3800562613 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed140f6510 msgr2=0x7fed140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:54.873 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.871+0000 7fed1c4cd700 1 --2- 172.21.15.178:0/3800562613 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed140f6510 0x7fed140f6930 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fed10009fb0 tx=0x7fed10009b30).stop 2023-11-15T04:55:54.874 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.871+0000 7fed1c4cd700 1 -- 172.21.15.178:0/3800562613 shutdown_connections 2023-11-15T04:55:54.874 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.871+0000 7fed1c4cd700 1 --2- 172.21.15.178:0/3800562613 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed140f6510 0x7fed140f6930 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:54.874 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.871+0000 7fed1c4cd700 1 -- 172.21.15.178:0/3800562613 >> 172.21.15.178:0/3800562613 conn(0x7fed140f1c70 msgr2=0x7fed140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:54.874 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.871+0000 7fed1c4cd700 1 -- 172.21.15.178:0/3800562613 shutdown_connections 2023-11-15T04:55:54.874 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.871+0000 7fed1c4cd700 1 -- 172.21.15.178:0/3800562613 wait complete. 2023-11-15T04:55:54.875 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.872+0000 7fed1c4cd700 1 Processor -- start 2023-11-15T04:55:54.875 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.872+0000 7fed1c4cd700 1 -- start start 2023-11-15T04:55:54.875 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.872+0000 7fed1c4cd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed140f6510 0x7fed14106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:54.875 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.872+0000 7fed1c4cd700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed14106c20 con 0x7fed140f6510 2023-11-15T04:55:54.875 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.872+0000 7fed1a269700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed140f6510 0x7fed14106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:54.876 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.872+0000 7fed1a269700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed140f6510 0x7fed14106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54208/0 (socket says 172.21.15.178:54208) 2023-11-15T04:55:54.876 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.872+0000 7fed1a269700 1 -- 172.21.15.178:0/2977648675 learned_addr learned my addr 172.21.15.178:0/2977648675 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:55:54.876 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.872+0000 7fed1a269700 1 -- 172.21.15.178:0/2977648675 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed10005040 con 0x7fed140f6510 2023-11-15T04:55:54.876 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.872+0000 7fed1a269700 1 --2- 172.21.15.178:0/2977648675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed140f6510 0x7fed14106730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fed10013510 tx=0x7fed1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:54.877 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.872+0000 7fed0affd700 1 -- 172.21.15.178:0/2977648675 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed10009320 con 0x7fed140f6510 2023-11-15T04:55:54.877 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.873+0000 7fed0affd700 1 -- 172.21.15.178:0/2977648675 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed10013f00 con 0x7fed140f6510 2023-11-15T04:55:54.878 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.873+0000 7fed1c4cd700 1 -- 172.21.15.178:0/2977648675 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed14106e20 con 0x7fed140f6510 2023-11-15T04:55:54.878 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.873+0000 7fed0affd700 1 -- 172.21.15.178:0/2977648675 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed10014d90 con 0x7fed140f6510 2023-11-15T04:55:54.878 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.873+0000 7fed1c4cd700 1 -- 172.21.15.178:0/2977648675 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed141072b0 con 0x7fed140f6510 2023-11-15T04:55:54.879 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.873+0000 7fed0affd700 1 -- 172.21.15.178:0/2977648675 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed10014420 con 0x7fed140f6510 2023-11-15T04:55:54.879 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.873+0000 7fed0affd700 1 --2- 172.21.15.178:0/2977648675 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fed0002ff60 0x7fed00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:54.880 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.873+0000 7fed1c4cd700 1 -- 172.21.15.178:0/2977648675 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed14045ad0 con 0x7fed140f6510 2023-11-15T04:55:54.880 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.874+0000 7fed0affd700 1 -- 172.21.15.178:0/2977648675 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed100410c0 con 0x7fed140f6510 2023-11-15T04:55:54.880 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.874+0000 7fed19a68700 1 --2- 172.21.15.178:0/2977648675 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fed0002ff60 0x7fed00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:54.881 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.874+0000 7fed19a68700 1 --2- 172.21.15.178:0/2977648675 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fed0002ff60 0x7fed00032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fed040096a0 tx=0x7fed04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:54.881 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:54.878+0000 7fed0affd700 1 -- 172.21.15.178:0/2977648675 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed1006d2b0 con 0x7fed140f6510 2023-11-15T04:55:55.050 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:55.049+0000 7fed1c4cd700 1 -- 172.21.15.178:0/2977648675 --> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi178", "target": ["mon-mgr", ""]}) v1 -- 0x7fed140f2e70 con 0x7fed0002ff60 2023-11-15T04:55:55.646 INFO:teuthology.orchestra.run.smithi178.stdout:Added host 'smithi178' 2023-11-15T04:55:55.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:55.644+0000 7fed0affd700 1 -- 172.21.15.178:0/2977648675 <== mgr.14162 v2:172.21.15.163:6800/3399151648 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fed140f2e70 con 0x7fed0002ff60 2023-11-15T04:55:55.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:55.647+0000 7fed1c4cd700 1 -- 172.21.15.178:0/2977648675 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fed0002ff60 msgr2=0x7fed00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:55.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:55.647+0000 7fed1c4cd700 1 --2- 172.21.15.178:0/2977648675 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fed0002ff60 0x7fed00032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fed040096a0 tx=0x7fed04006b40).stop 2023-11-15T04:55:55.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:55.647+0000 7fed1c4cd700 1 -- 172.21.15.178:0/2977648675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed140f6510 msgr2=0x7fed14106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:55.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:55.647+0000 7fed1c4cd700 1 --2- 172.21.15.178:0/2977648675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed140f6510 0x7fed14106730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fed10013510 tx=0x7fed1000b040).stop 2023-11-15T04:55:55.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:55.647+0000 7fed1c4cd700 1 -- 172.21.15.178:0/2977648675 shutdown_connections 2023-11-15T04:55:55.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:55.647+0000 7fed1c4cd700 1 --2- 172.21.15.178:0/2977648675 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fed0002ff60 0x7fed00032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:55.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:55.647+0000 7fed1c4cd700 1 --2- 172.21.15.178:0/2977648675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed140f6510 0x7fed14106730 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:55.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:55.647+0000 7fed1c4cd700 1 -- 172.21.15.178:0/2977648675 >> 172.21.15.178:0/2977648675 conn(0x7fed140f1c70 msgr2=0x7fed140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:55.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:55.647+0000 7fed1c4cd700 1 -- 172.21.15.178:0/2977648675 shutdown_connections 2023-11-15T04:55:55.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:55.647+0000 7fed1c4cd700 1 -- 172.21.15.178:0/2977648675 wait complete. 2023-11-15T04:55:55.978 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph orch host ls --format=json 2023-11-15T04:55:56.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:55 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:54.312045+0000 mgr.smithi163.qesyrx (mgr.14162) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:55:56.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:55 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:55.050940+0000 mgr.smithi163.qesyrx (mgr.14162) 12 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi178", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:55:56.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:55 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:55.644369+0000 mon.smithi163 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' 2023-11-15T04:55:57.132 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.131+0000 7f037dcfe700 1 -- 172.21.15.178:0/467231061 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03780601e0 msgr2=0x7f03780f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:57.133 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.131+0000 7f037dcfe700 1 --2- 172.21.15.178:0/467231061 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03780601e0 0x7f03780f8f20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f036c009fb0 tx=0x7f036c009b30).stop 2023-11-15T04:55:57.133 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.131+0000 7f037dcfe700 1 -- 172.21.15.178:0/467231061 shutdown_connections 2023-11-15T04:55:57.133 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.131+0000 7f037dcfe700 1 --2- 172.21.15.178:0/467231061 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03780601e0 0x7f03780f8f20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:57.133 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.131+0000 7f037dcfe700 1 -- 172.21.15.178:0/467231061 >> 172.21.15.178:0/467231061 conn(0x7f03780f09a0 msgr2=0x7f03780f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:57.134 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.131+0000 7f037dcfe700 1 -- 172.21.15.178:0/467231061 shutdown_connections 2023-11-15T04:55:57.134 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.131+0000 7f037dcfe700 1 -- 172.21.15.178:0/467231061 wait complete. 2023-11-15T04:55:57.134 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.132+0000 7f037dcfe700 1 Processor -- start 2023-11-15T04:55:57.134 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.132+0000 7f037dcfe700 1 -- start start 2023-11-15T04:55:57.134 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.132+0000 7f037dcfe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03780601e0 0x7f0378106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:57.135 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.132+0000 7f037dcfe700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03781069f0 con 0x7f03780601e0 2023-11-15T04:55:57.135 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.132+0000 7f037ccfc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03780601e0 0x7f0378106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:57.135 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.132+0000 7f037ccfc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03780601e0 0x7f0378106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54234/0 (socket says 172.21.15.178:54234) 2023-11-15T04:55:57.135 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.132+0000 7f037ccfc700 1 -- 172.21.15.178:0/1258391760 learned_addr learned my addr 172.21.15.178:0/1258391760 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:55:57.137 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.133+0000 7f037ccfc700 1 -- 172.21.15.178:0/1258391760 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036c005040 con 0x7f03780601e0 2023-11-15T04:55:57.137 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.133+0000 7f037ccfc700 1 --2- 172.21.15.178:0/1258391760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03780601e0 0x7f0378106500 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f036c013510 tx=0x7f036c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:57.137 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.133+0000 7f03757fa700 1 -- 172.21.15.178:0/1258391760 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f036c009320 con 0x7f03780601e0 2023-11-15T04:55:57.137 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.133+0000 7f03757fa700 1 -- 172.21.15.178:0/1258391760 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f036c013f00 con 0x7f03780601e0 2023-11-15T04:55:57.138 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.133+0000 7f037dcfe700 1 -- 172.21.15.178:0/1258391760 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0378106bf0 con 0x7f03780601e0 2023-11-15T04:55:57.138 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.133+0000 7f03757fa700 1 -- 172.21.15.178:0/1258391760 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f036c014d90 con 0x7f03780601e0 2023-11-15T04:55:57.138 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.133+0000 7f037dcfe700 1 -- 172.21.15.178:0/1258391760 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0378107080 con 0x7f03780601e0 2023-11-15T04:55:57.138 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.134+0000 7f03757fa700 1 -- 172.21.15.178:0/1258391760 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f036c014420 con 0x7f03780601e0 2023-11-15T04:55:57.138 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.134+0000 7f037dcfe700 1 -- 172.21.15.178:0/1258391760 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0378045ad0 con 0x7f03780601e0 2023-11-15T04:55:57.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.134+0000 7f03757fa700 1 --2- 172.21.15.178:0/1258391760 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f036802ff60 0x7f0368032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:57.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.134+0000 7f03757fa700 1 -- 172.21.15.178:0/1258391760 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f036c040ff0 con 0x7f03780601e0 2023-11-15T04:55:57.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.134+0000 7f0377fff700 1 --2- 172.21.15.178:0/1258391760 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f036802ff60 0x7f0368032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:57.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.135+0000 7f0377fff700 1 --2- 172.21.15.178:0/1258391760 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f036802ff60 0x7f0368032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f03640096a0 tx=0x7f0364006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:57.140 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.139+0000 7f03757fa700 1 -- 172.21.15.178:0/1258391760 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f036c0095f0 con 0x7f03780601e0 2023-11-15T04:55:57.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:56 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cephadm 2023-11-15T04:55:55.644788+0000 mgr.smithi163.qesyrx (mgr.14162) 13 : cephadm [INF] Added host smithi178 2023-11-15T04:55:57.309 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.309+0000 7f037dcfe700 1 -- 172.21.15.178:0/1258391760 --> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f03780f1b30 con 0x7f036802ff60 2023-11-15T04:55:57.311 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.311+0000 7f03757fa700 1 -- 172.21.15.178:0/1258391760 <== mgr.14162 v2:172.21.15.163:6800/3399151648 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+157 (secure 0 0 0) 0x7f03780f1b30 con 0x7f036802ff60 2023-11-15T04:55:57.311 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:55:57.312 INFO:teuthology.orchestra.run.smithi178.stdout:[{"addr": "172.21.15.163", "hostname": "smithi163", "labels": [], "status": ""}, {"addr": "smithi178", "hostname": "smithi178", "labels": [], "status": ""}] 2023-11-15T04:55:57.313 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.312+0000 7f037dcfe700 1 -- 172.21.15.178:0/1258391760 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f036802ff60 msgr2=0x7f0368032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:57.313 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.312+0000 7f037dcfe700 1 --2- 172.21.15.178:0/1258391760 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f036802ff60 0x7f0368032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f03640096a0 tx=0x7f0364006b40).stop 2023-11-15T04:55:57.313 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.312+0000 7f037dcfe700 1 -- 172.21.15.178:0/1258391760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03780601e0 msgr2=0x7f0378106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:57.314 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.312+0000 7f037dcfe700 1 --2- 172.21.15.178:0/1258391760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03780601e0 0x7f0378106500 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f036c013510 tx=0x7f036c00b040).stop 2023-11-15T04:55:57.314 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.312+0000 7f037dcfe700 1 -- 172.21.15.178:0/1258391760 shutdown_connections 2023-11-15T04:55:57.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.312+0000 7f037dcfe700 1 --2- 172.21.15.178:0/1258391760 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f036802ff60 0x7f0368032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:57.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.312+0000 7f037dcfe700 1 --2- 172.21.15.178:0/1258391760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03780601e0 0x7f0378106500 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:57.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.312+0000 7f037dcfe700 1 -- 172.21.15.178:0/1258391760 >> 172.21.15.178:0/1258391760 conn(0x7f03780f09a0 msgr2=0x7f03780f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:57.316 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.312+0000 7f037dcfe700 1 -- 172.21.15.178:0/1258391760 shutdown_connections 2023-11-15T04:55:57.316 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:55:57.312+0000 7f037dcfe700 1 -- 172.21.15.178:0/1258391760 wait complete. 2023-11-15T04:55:57.684 INFO:tasks.cephadm:Setting crush tunables to default 2023-11-15T04:55:57.685 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph osd crush tunables default 2023-11-15T04:55:57.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:57 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:56.312385+0000 mgr.smithi163.qesyrx (mgr.14162) 14 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:55:58.806 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.803+0000 7fa1fcb1e700 1 -- 172.21.15.163:0/3766896974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1f80f62a0 msgr2=0x7fa1f80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:58.806 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.803+0000 7fa1fcb1e700 1 --2- 172.21.15.163:0/3766896974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1f80f62a0 0x7fa1f80f66c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fa1ec009fb0 tx=0x7fa1ec009b30).stop 2023-11-15T04:55:58.807 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.803+0000 7fa1fcb1e700 1 -- 172.21.15.163:0/3766896974 shutdown_connections 2023-11-15T04:55:58.807 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.803+0000 7fa1fcb1e700 1 --2- 172.21.15.163:0/3766896974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1f80f62a0 0x7fa1f80f66c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:58.807 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.803+0000 7fa1fcb1e700 1 -- 172.21.15.163:0/3766896974 >> 172.21.15.163:0/3766896974 conn(0x7fa1f80f1aa0 msgr2=0x7fa1f80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:58.807 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.803+0000 7fa1fcb1e700 1 -- 172.21.15.163:0/3766896974 shutdown_connections 2023-11-15T04:55:58.808 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.803+0000 7fa1fcb1e700 1 -- 172.21.15.163:0/3766896974 wait complete. 2023-11-15T04:55:58.808 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.804+0000 7fa1fcb1e700 1 Processor -- start 2023-11-15T04:55:58.808 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.804+0000 7fa1fcb1e700 1 -- start start 2023-11-15T04:55:58.808 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.804+0000 7fa1fcb1e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1f80f62a0 0x7fa1f81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:58.808 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.804+0000 7fa1fcb1e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f81069a0 con 0x7fa1f80f62a0 2023-11-15T04:55:58.808 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.804+0000 7fa1f77fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1f80f62a0 0x7fa1f81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:58.809 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.804+0000 7fa1f77fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1f80f62a0 0x7fa1f81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:48522/0 (socket says 172.21.15.163:48522) 2023-11-15T04:55:58.809 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.804+0000 7fa1f77fe700 1 -- 172.21.15.163:0/1342350257 learned_addr learned my addr 172.21.15.163:0/1342350257 (peer_addr_for_me v2:172.21.15.163:0/0) 2023-11-15T04:55:58.809 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.804+0000 7fa1f77fe700 1 -- 172.21.15.163:0/1342350257 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ec005040 con 0x7fa1f80f62a0 2023-11-15T04:55:58.809 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.804+0000 7fa1f77fe700 1 --2- 172.21.15.163:0/1342350257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1f80f62a0 0x7fa1f81064b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fa1ec013510 tx=0x7fa1ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:58.809 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.805+0000 7fa1dffff700 1 -- 172.21.15.163:0/1342350257 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1ec009320 con 0x7fa1f80f62a0 2023-11-15T04:55:58.810 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.805+0000 7fa1dffff700 1 -- 172.21.15.163:0/1342350257 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa1ec013f00 con 0x7fa1f80f62a0 2023-11-15T04:55:58.810 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.805+0000 7fa1fcb1e700 1 -- 172.21.15.163:0/1342350257 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f8106ba0 con 0x7fa1f80f62a0 2023-11-15T04:55:58.810 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.805+0000 7fa1dffff700 1 -- 172.21.15.163:0/1342350257 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1ec014dc0 con 0x7fa1f80f62a0 2023-11-15T04:55:58.810 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.805+0000 7fa1fcb1e700 1 -- 172.21.15.163:0/1342350257 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f8107030 con 0x7fa1f80f62a0 2023-11-15T04:55:58.811 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.805+0000 7fa1dffff700 1 -- 172.21.15.163:0/1342350257 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1ec0263a0 con 0x7fa1f80f62a0 2023-11-15T04:55:58.812 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.805+0000 7fa1dffff700 1 --2- 172.21.15.163:0/1342350257 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa1e002ffb0 0x7fa1e0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:55:58.812 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.805+0000 7fa1fcb1e700 1 -- 172.21.15.163:0/1342350257 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f8045ad0 con 0x7fa1f80f62a0 2023-11-15T04:55:58.812 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.805+0000 7fa1dffff700 1 -- 172.21.15.163:0/1342350257 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa1ec040ff0 con 0x7fa1f80f62a0 2023-11-15T04:55:58.812 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.806+0000 7fa1f6ffd700 1 --2- 172.21.15.163:0/1342350257 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa1e002ffb0 0x7fa1e0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:55:58.812 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.806+0000 7fa1f6ffd700 1 --2- 172.21.15.163:0/1342350257 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa1e002ffb0 0x7fa1e0032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fa1e80096a0 tx=0x7fa1e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:55:58.813 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.810+0000 7fa1dffff700 1 -- 172.21.15.163:0/1342350257 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa1ec012920 con 0x7fa1f80f62a0 2023-11-15T04:55:58.952 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:58 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:57.310810+0000 mgr.smithi163.qesyrx (mgr.14162) 15 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-15T04:55:58.966 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:58.965+0000 7fa1fcb1e700 1 -- 172.21.15.163:0/1342350257 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fa1f8107ee0 con 0x7fa1f80f62a0 2023-11-15T04:55:59.757 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:59.755+0000 7fa1dffff700 1 -- 172.21.15.163:0/1342350257 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7fa1ec02b030 con 0x7fa1f80f62a0 2023-11-15T04:55:59.759 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:59.758+0000 7fa1fcb1e700 1 -- 172.21.15.163:0/1342350257 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa1e002ffb0 msgr2=0x7fa1e0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:59.760 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:59.758+0000 7fa1fcb1e700 1 --2- 172.21.15.163:0/1342350257 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa1e002ffb0 0x7fa1e0032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fa1e80096a0 tx=0x7fa1e8006b40).stop 2023-11-15T04:55:59.760 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:59.758+0000 7fa1fcb1e700 1 -- 172.21.15.163:0/1342350257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1f80f62a0 msgr2=0x7fa1f81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:55:59.761 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:59.758+0000 7fa1fcb1e700 1 --2- 172.21.15.163:0/1342350257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1f80f62a0 0x7fa1f81064b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fa1ec013510 tx=0x7fa1ec00b040).stop 2023-11-15T04:55:59.761 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:59.758+0000 7fa1fcb1e700 1 -- 172.21.15.163:0/1342350257 shutdown_connections 2023-11-15T04:55:59.761 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:59.758+0000 7fa1fcb1e700 1 --2- 172.21.15.163:0/1342350257 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa1e002ffb0 0x7fa1e0032470 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:59.762 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:59.758+0000 7fa1fcb1e700 1 --2- 172.21.15.163:0/1342350257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1f80f62a0 0x7fa1f81064b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:55:59.762 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:59.758+0000 7fa1fcb1e700 1 -- 172.21.15.163:0/1342350257 >> 172.21.15.163:0/1342350257 conn(0x7fa1f80f1aa0 msgr2=0x7fa1f80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:55:59.762 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:59.758+0000 7fa1fcb1e700 1 -- 172.21.15.163:0/1342350257 shutdown_connections 2023-11-15T04:55:59.762 INFO:teuthology.orchestra.run.smithi163.stderr:2023-11-15T04:55:59.758+0000 7fa1fcb1e700 1 -- 172.21.15.163:0/1342350257 wait complete. 2023-11-15T04:55:59.763 INFO:teuthology.orchestra.run.smithi163.stderr:adjusted tunables profile to default 2023-11-15T04:56:00.044 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:59 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:58.312646+0000 mgr.smithi163.qesyrx (mgr.14162) 16 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:00.044 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:55:59 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:58.967412+0000 mon.smithi163 (mon.0) 117 : audit [INF] from='client.? 172.21.15.163:0/1342350257' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-15T04:56:00.089 INFO:tasks.cephadm:Adding mon.smithi163 on smithi163 2023-11-15T04:56:00.089 INFO:tasks.cephadm:Adding mon.smithi178 on smithi178 2023-11-15T04:56:00.090 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph orch apply mon '2;smithi163:172.21.15.163=smithi163;smithi178:172.21.15.178=smithi178' 2023-11-15T04:56:01.174 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.173+0000 7fdd8bfff700 1 -- 172.21.15.178:0/1499514260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0f51a0 msgr2=0x7fdd8c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:01.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.173+0000 7fdd8bfff700 1 --2- 172.21.15.178:0/1499514260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0f51a0 0x7fdd8c0f55c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fdd7c009fb0 tx=0x7fdd7c009b30).stop 2023-11-15T04:56:01.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.173+0000 7fdd8bfff700 1 -- 172.21.15.178:0/1499514260 shutdown_connections 2023-11-15T04:56:01.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.173+0000 7fdd8bfff700 1 --2- 172.21.15.178:0/1499514260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0f51a0 0x7fdd8c0f55c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:01.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.173+0000 7fdd8bfff700 1 -- 172.21.15.178:0/1499514260 >> 172.21.15.178:0/1499514260 conn(0x7fdd8c0f09a0 msgr2=0x7fdd8c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:01.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.173+0000 7fdd8bfff700 1 -- 172.21.15.178:0/1499514260 shutdown_connections 2023-11-15T04:56:01.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.173+0000 7fdd8bfff700 1 -- 172.21.15.178:0/1499514260 wait complete. 2023-11-15T04:56:01.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.173+0000 7fdd8bfff700 1 Processor -- start 2023-11-15T04:56:01.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.173+0000 7fdd8bfff700 1 -- start start 2023-11-15T04:56:01.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.173+0000 7fdd8bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0f51a0 0x7fdd8c1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:01.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.173+0000 7fdd8bfff700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c060220 con 0x7fdd8c0f51a0 2023-11-15T04:56:01.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.174+0000 7fdd8affd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0f51a0 0x7fdd8c1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:01.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.174+0000 7fdd8affd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0f51a0 0x7fdd8c1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54262/0 (socket says 172.21.15.178:54262) 2023-11-15T04:56:01.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.174+0000 7fdd8affd700 1 -- 172.21.15.178:0/2784278833 learned_addr learned my addr 172.21.15.178:0/2784278833 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:56:01.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.174+0000 7fdd8affd700 1 -- 172.21.15.178:0/2784278833 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7c005040 con 0x7fdd8c0f51a0 2023-11-15T04:56:01.178 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.174+0000 7fdd8affd700 1 --2- 172.21.15.178:0/2784278833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0f51a0 0x7fdd8c1091d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fdd7c013510 tx=0x7fdd7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:01.178 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.174+0000 7fdd737fe700 1 -- 172.21.15.178:0/2784278833 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd7c009320 con 0x7fdd8c0f51a0 2023-11-15T04:56:01.178 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.174+0000 7fdd737fe700 1 -- 172.21.15.178:0/2784278833 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd7c013f00 con 0x7fdd8c0f51a0 2023-11-15T04:56:01.178 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.174+0000 7fdd8bfff700 1 -- 172.21.15.178:0/2784278833 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd8c060420 con 0x7fdd8c0f51a0 2023-11-15T04:56:01.179 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.174+0000 7fdd737fe700 1 -- 172.21.15.178:0/2784278833 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd7c014dc0 con 0x7fdd8c0f51a0 2023-11-15T04:56:01.179 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.174+0000 7fdd8bfff700 1 -- 172.21.15.178:0/2784278833 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd8c109880 con 0x7fdd8c0f51a0 2023-11-15T04:56:01.181 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.175+0000 7fdd737fe700 1 -- 172.21.15.178:0/2784278833 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd7c014420 con 0x7fdd8c0f51a0 2023-11-15T04:56:01.181 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.175+0000 7fdd737fe700 1 --2- 172.21.15.178:0/2784278833 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdd7402ffb0 0x7fdd74032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:01.181 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.175+0000 7fdd737fe700 1 -- 172.21.15.178:0/2784278833 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd7c040ff0 con 0x7fdd8c0f51a0 2023-11-15T04:56:01.181 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.175+0000 7fdd8bfff700 1 -- 172.21.15.178:0/2784278833 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd8c059100 con 0x7fdd8c0f51a0 2023-11-15T04:56:01.181 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.175+0000 7fdd8a7fc700 1 --2- 172.21.15.178:0/2784278833 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdd7402ffb0 0x7fdd74032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:01.182 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.176+0000 7fdd8a7fc700 1 --2- 172.21.15.178:0/2784278833 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdd7402ffb0 0x7fdd74032470 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fdd800096a0 tx=0x7fdd80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:01.182 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.180+0000 7fdd737fe700 1 -- 172.21.15.178:0/2784278833 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd7c0095f0 con 0x7fdd8c0f51a0 2023-11-15T04:56:01.204 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:00 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:55:59.757229+0000 mon.smithi163 (mon.0) 118 : audit [INF] from='client.? 172.21.15.163:0/1342350257' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-15T04:56:01.205 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:00 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:55:59.757292+0000 mon.smithi163 (mon.0) 119 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-11-15T04:56:01.356 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.356+0000 7fdd8bfff700 1 -- 172.21.15.178:0/2784278833 --> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi163:172.21.15.163=smithi163;smithi178:172.21.15.178=smithi178", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd8c0f1c20 con 0x7fdd7402ffb0 2023-11-15T04:56:01.362 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.361+0000 7fdd737fe700 1 -- 172.21.15.178:0/2784278833 <== mgr.14162 v2:172.21.15.163:6800/3399151648 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fdd8c0f1c20 con 0x7fdd7402ffb0 2023-11-15T04:56:01.362 INFO:teuthology.orchestra.run.smithi178.stdout:Scheduled mon update... 2023-11-15T04:56:01.363 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.363+0000 7fdd8bfff700 1 -- 172.21.15.178:0/2784278833 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdd7402ffb0 msgr2=0x7fdd74032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:01.364 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.363+0000 7fdd8bfff700 1 --2- 172.21.15.178:0/2784278833 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdd7402ffb0 0x7fdd74032470 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fdd800096a0 tx=0x7fdd80006b40).stop 2023-11-15T04:56:01.364 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.363+0000 7fdd8bfff700 1 -- 172.21.15.178:0/2784278833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0f51a0 msgr2=0x7fdd8c1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:01.364 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.363+0000 7fdd8bfff700 1 --2- 172.21.15.178:0/2784278833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0f51a0 0x7fdd8c1091d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fdd7c013510 tx=0x7fdd7c00b040).stop 2023-11-15T04:56:01.364 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.363+0000 7fdd8bfff700 1 -- 172.21.15.178:0/2784278833 shutdown_connections 2023-11-15T04:56:01.364 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.363+0000 7fdd8bfff700 1 --2- 172.21.15.178:0/2784278833 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdd7402ffb0 0x7fdd74032470 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:01.365 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.363+0000 7fdd8bfff700 1 --2- 172.21.15.178:0/2784278833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0f51a0 0x7fdd8c1091d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:01.365 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.363+0000 7fdd8bfff700 1 -- 172.21.15.178:0/2784278833 >> 172.21.15.178:0/2784278833 conn(0x7fdd8c0f09a0 msgr2=0x7fdd8c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:01.365 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.363+0000 7fdd8bfff700 1 -- 172.21.15.178:0/2784278833 shutdown_connections 2023-11-15T04:56:01.365 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:01.363+0000 7fdd8bfff700 1 -- 172.21.15.178:0/2784278833 wait complete. 2023-11-15T04:56:01.745 DEBUG:teuthology.orchestra.run.smithi178:mon.smithi178> sudo journalctl -f -n 0 -u ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mon.smithi178.service 2023-11-15T04:56:01.749 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:56:01.749 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:56:01.777 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:-- Logs begin at Wed 2023-11-15 04:41:55 UTC. -- 2023-11-15T04:56:02.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:01 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:00.312976+0000 mgr.smithi163.qesyrx (mgr.14162) 17 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:02.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:01 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:01.361660+0000 mon.smithi163 (mon.0) 120 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' 2023-11-15T04:56:02.960 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.958+0000 7fd363b98700 1 -- 172.21.15.178:0/3355527335 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd35c0f5410 msgr2=0x7fd35c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:02.961 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.958+0000 7fd363b98700 1 --2- 172.21.15.178:0/3355527335 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd35c0f5410 0x7fd35c0f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fd34c009fb0 tx=0x7fd34c009b30).stop 2023-11-15T04:56:02.961 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.958+0000 7fd363b98700 1 -- 172.21.15.178:0/3355527335 shutdown_connections 2023-11-15T04:56:02.961 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.958+0000 7fd363b98700 1 --2- 172.21.15.178:0/3355527335 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd35c0f5410 0x7fd35c0f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:02.962 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.958+0000 7fd363b98700 1 -- 172.21.15.178:0/3355527335 >> 172.21.15.178:0/3355527335 conn(0x7fd35c0f0b70 msgr2=0x7fd35c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:02.962 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.958+0000 7fd363b98700 1 -- 172.21.15.178:0/3355527335 shutdown_connections 2023-11-15T04:56:02.962 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.958+0000 7fd363b98700 1 -- 172.21.15.178:0/3355527335 wait complete. 2023-11-15T04:56:02.962 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.959+0000 7fd363b98700 1 Processor -- start 2023-11-15T04:56:02.962 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.959+0000 7fd363b98700 1 -- start start 2023-11-15T04:56:02.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.959+0000 7fd363b98700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd35c0f5410 0x7fd35c109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:02.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.959+0000 7fd363b98700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c060240 con 0x7fd35c0f5410 2023-11-15T04:56:02.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.959+0000 7fd361934700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd35c0f5410 0x7fd35c109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:02.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.959+0000 7fd361934700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd35c0f5410 0x7fd35c109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54272/0 (socket says 172.21.15.178:54272) 2023-11-15T04:56:02.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.959+0000 7fd361934700 1 -- 172.21.15.178:0/2163012202 learned_addr learned my addr 172.21.15.178:0/2163012202 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:56:02.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.960+0000 7fd361934700 1 -- 172.21.15.178:0/2163012202 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c005040 con 0x7fd35c0f5410 2023-11-15T04:56:02.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.960+0000 7fd361934700 1 --2- 172.21.15.178:0/2163012202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd35c0f5410 0x7fd35c109400 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fd34c014000 tx=0x7fd34c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:02.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.960+0000 7fd3527fc700 1 -- 172.21.15.178:0/2163012202 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd34c0095f0 con 0x7fd35c0f5410 2023-11-15T04:56:02.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.960+0000 7fd3527fc700 1 -- 172.21.15.178:0/2163012202 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd34c007e70 con 0x7fd35c0f5410 2023-11-15T04:56:02.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.960+0000 7fd363b98700 1 -- 172.21.15.178:0/2163012202 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35c060440 con 0x7fd35c0f5410 2023-11-15T04:56:02.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.960+0000 7fd3527fc700 1 -- 172.21.15.178:0/2163012202 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd34c014ea0 con 0x7fd35c0f5410 2023-11-15T04:56:02.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.960+0000 7fd363b98700 1 -- 172.21.15.178:0/2163012202 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35c109ab0 con 0x7fd35c0f5410 2023-11-15T04:56:02.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.961+0000 7fd3527fc700 1 -- 172.21.15.178:0/2163012202 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd34c014690 con 0x7fd35c0f5410 2023-11-15T04:56:02.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.961+0000 7fd363b98700 1 -- 172.21.15.178:0/2163012202 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd35c045ad0 con 0x7fd35c0f5410 2023-11-15T04:56:02.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.961+0000 7fd3527fc700 1 --2- 172.21.15.178:0/2163012202 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd34802ff60 0x7fd348032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:02.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.961+0000 7fd3527fc700 1 -- 172.21.15.178:0/2163012202 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd34c040870 con 0x7fd35c0f5410 2023-11-15T04:56:02.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.961+0000 7fd361133700 1 --2- 172.21.15.178:0/2163012202 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd34802ff60 0x7fd348032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:02.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.962+0000 7fd361133700 1 --2- 172.21.15.178:0/2163012202 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd34802ff60 0x7fd348032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fd3580096a0 tx=0x7fd358006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:02.968 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:02.966+0000 7fd3527fc700 1 -- 172.21.15.178:0/2163012202 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd34c00f300 con 0x7fd35c0f5410 2023-11-15T04:56:03.174 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:03.173+0000 7fd363b98700 1 -- 172.21.15.178:0/2163012202 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd35c0fae60 con 0x7fd35c0f5410 2023-11-15T04:56:03.174 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:03.174+0000 7fd3527fc700 1 -- 172.21.15.178:0/2163012202 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd34c00f300 con 0x7fd35c0f5410 2023-11-15T04:56:03.175 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:56:03.175 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:56:03.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:03.176+0000 7fd363b98700 1 -- 172.21.15.178:0/2163012202 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd34802ff60 msgr2=0x7fd348032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:03.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:03.176+0000 7fd363b98700 1 --2- 172.21.15.178:0/2163012202 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd34802ff60 0x7fd348032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fd3580096a0 tx=0x7fd358006b40).stop 2023-11-15T04:56:03.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:03.176+0000 7fd363b98700 1 -- 172.21.15.178:0/2163012202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd35c0f5410 msgr2=0x7fd35c109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:03.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:03.176+0000 7fd363b98700 1 --2- 172.21.15.178:0/2163012202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd35c0f5410 0x7fd35c109400 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fd34c014000 tx=0x7fd34c00b040).stop 2023-11-15T04:56:03.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:03.176+0000 7fd363b98700 1 -- 172.21.15.178:0/2163012202 shutdown_connections 2023-11-15T04:56:03.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:03.176+0000 7fd363b98700 1 --2- 172.21.15.178:0/2163012202 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd34802ff60 0x7fd348032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:03.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:03.176+0000 7fd363b98700 1 --2- 172.21.15.178:0/2163012202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd35c0f5410 0x7fd35c109400 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:03.178 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:03.176+0000 7fd363b98700 1 -- 172.21.15.178:0/2163012202 >> 172.21.15.178:0/2163012202 conn(0x7fd35c0f0b70 msgr2=0x7fd35c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:03.178 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:03.176+0000 7fd363b98700 1 -- 172.21.15.178:0/2163012202 shutdown_connections 2023-11-15T04:56:03.178 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:03.176+0000 7fd363b98700 1 -- 172.21.15.178:0/2163012202 wait complete. 2023-11-15T04:56:03.178 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:56:03.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:02 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:01.357663+0000 mgr.smithi163.qesyrx (mgr.14162) 18 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi163:172.21.15.163=smithi163;smithi178:172.21.15.178=smithi178", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T04:56:03.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:02 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cephadm 2023-11-15T04:56:01.359062+0000 mgr.smithi163.qesyrx (mgr.14162) 19 : cephadm [INF] Saving service mon spec with placement smithi163:172.21.15.163=smithi163;smithi178:172.21.15.178=smithi178;count:2 2023-11-15T04:56:04.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:03 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:02.313278+0000 mgr.smithi163.qesyrx (mgr.14162) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:04.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:03 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:03.174702+0000 mon.smithi163 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.178:0/2163012202' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:56:04.545 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:56:04.545 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:56:05.727 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.725+0000 7fd66d1f3700 1 -- 172.21.15.178:0/3806733991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6680f78e0 msgr2=0x7fd6680f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:05.727 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.725+0000 7fd66d1f3700 1 --2- 172.21.15.178:0/3806733991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6680f78e0 0x7fd6680f7d00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fd658009fb0 tx=0x7fd658009b30).stop 2023-11-15T04:56:05.728 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.725+0000 7fd66d1f3700 1 -- 172.21.15.178:0/3806733991 shutdown_connections 2023-11-15T04:56:05.728 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.725+0000 7fd66d1f3700 1 --2- 172.21.15.178:0/3806733991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6680f78e0 0x7fd6680f7d00 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:05.728 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.725+0000 7fd66d1f3700 1 -- 172.21.15.178:0/3806733991 >> 172.21.15.178:0/3806733991 conn(0x7fd6680f0120 msgr2=0x7fd6680f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:05.728 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.725+0000 7fd66d1f3700 1 -- 172.21.15.178:0/3806733991 shutdown_connections 2023-11-15T04:56:05.728 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.725+0000 7fd66d1f3700 1 -- 172.21.15.178:0/3806733991 wait complete. 2023-11-15T04:56:05.729 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.726+0000 7fd66d1f3700 1 Processor -- start 2023-11-15T04:56:05.729 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.726+0000 7fd66d1f3700 1 -- start start 2023-11-15T04:56:05.729 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.726+0000 7fd66d1f3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6680f78e0 0x7fd6681064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:05.729 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.726+0000 7fd66d1f3700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6681069d0 con 0x7fd6680f78e0 2023-11-15T04:56:05.729 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.726+0000 7fd667fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6680f78e0 0x7fd6681064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:05.729 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.726+0000 7fd667fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6680f78e0 0x7fd6681064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:48950/0 (socket says 172.21.15.178:48950) 2023-11-15T04:56:05.730 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.726+0000 7fd667fff700 1 -- 172.21.15.178:0/3105819467 learned_addr learned my addr 172.21.15.178:0/3105819467 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:56:05.730 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.726+0000 7fd667fff700 1 -- 172.21.15.178:0/3105819467 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd658005040 con 0x7fd6680f78e0 2023-11-15T04:56:05.730 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.727+0000 7fd667fff700 1 --2- 172.21.15.178:0/3105819467 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6680f78e0 0x7fd6681064e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fd658013510 tx=0x7fd65800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:05.730 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.727+0000 7fd664ff9700 1 -- 172.21.15.178:0/3105819467 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd658009320 con 0x7fd6680f78e0 2023-11-15T04:56:05.730 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.727+0000 7fd664ff9700 1 -- 172.21.15.178:0/3105819467 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd658013f00 con 0x7fd6680f78e0 2023-11-15T04:56:05.731 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.727+0000 7fd66d1f3700 1 -- 172.21.15.178:0/3105819467 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd668106bd0 con 0x7fd6680f78e0 2023-11-15T04:56:05.731 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.727+0000 7fd664ff9700 1 -- 172.21.15.178:0/3105819467 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd658014dc0 con 0x7fd6680f78e0 2023-11-15T04:56:05.731 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.727+0000 7fd66d1f3700 1 -- 172.21.15.178:0/3105819467 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd668107060 con 0x7fd6680f78e0 2023-11-15T04:56:05.732 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.728+0000 7fd664ff9700 1 -- 172.21.15.178:0/3105819467 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd65801d3d0 con 0x7fd6680f78e0 2023-11-15T04:56:05.732 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.728+0000 7fd664ff9700 1 --2- 172.21.15.178:0/3105819467 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd65002ff60 0x7fd650032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:05.732 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.728+0000 7fd664ff9700 1 -- 172.21.15.178:0/3105819467 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd658041030 con 0x7fd6680f78e0 2023-11-15T04:56:05.733 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.728+0000 7fd66d1f3700 1 -- 172.21.15.178:0/3105819467 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd668045ad0 con 0x7fd6680f78e0 2023-11-15T04:56:05.733 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.728+0000 7fd6677fe700 1 --2- 172.21.15.178:0/3105819467 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd65002ff60 0x7fd650032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:05.733 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.728+0000 7fd6677fe700 1 --2- 172.21.15.178:0/3105819467 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd65002ff60 0x7fd650032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fd65c0096a0 tx=0x7fd65c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:05.733 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.733+0000 7fd664ff9700 1 -- 172.21.15.178:0/3105819467 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd658026490 con 0x7fd6680f78e0 2023-11-15T04:56:05.942 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.940+0000 7fd66d1f3700 1 -- 172.21.15.178:0/3105819467 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd668107da0 con 0x7fd6680f78e0 2023-11-15T04:56:05.942 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.941+0000 7fd664ff9700 1 -- 172.21.15.178:0/3105819467 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd6580126f0 con 0x7fd6680f78e0 2023-11-15T04:56:05.943 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:56:05.943 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:56:05.945 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.943+0000 7fd66d1f3700 1 -- 172.21.15.178:0/3105819467 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd65002ff60 msgr2=0x7fd650032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:05.945 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.943+0000 7fd66d1f3700 1 --2- 172.21.15.178:0/3105819467 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd65002ff60 0x7fd650032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fd65c0096a0 tx=0x7fd65c006b40).stop 2023-11-15T04:56:05.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.943+0000 7fd66d1f3700 1 -- 172.21.15.178:0/3105819467 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6680f78e0 msgr2=0x7fd6681064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:05.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.943+0000 7fd66d1f3700 1 --2- 172.21.15.178:0/3105819467 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6680f78e0 0x7fd6681064e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fd658013510 tx=0x7fd65800b040).stop 2023-11-15T04:56:05.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.943+0000 7fd66d1f3700 1 -- 172.21.15.178:0/3105819467 shutdown_connections 2023-11-15T04:56:05.947 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.943+0000 7fd66d1f3700 1 --2- 172.21.15.178:0/3105819467 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd65002ff60 0x7fd650032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:05.947 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.943+0000 7fd66d1f3700 1 --2- 172.21.15.178:0/3105819467 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6680f78e0 0x7fd6681064e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:05.947 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.943+0000 7fd66d1f3700 1 -- 172.21.15.178:0/3105819467 >> 172.21.15.178:0/3105819467 conn(0x7fd6680f0120 msgr2=0x7fd6680f0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:05.948 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.943+0000 7fd66d1f3700 1 -- 172.21.15.178:0/3105819467 shutdown_connections 2023-11-15T04:56:05.948 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:05.943+0000 7fd66d1f3700 1 -- 172.21.15.178:0/3105819467 wait complete. 2023-11-15T04:56:05.948 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:56:06.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:05 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:04.313569+0000 mgr.smithi163.qesyrx (mgr.14162) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:07.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:06 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:05.941774+0000 mon.smithi163 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.178:0/3105819467' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:56:07.323 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:56:07.324 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:56:08.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:07 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:06.313937+0000 mgr.smithi163.qesyrx (mgr.14162) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:08.481 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.479+0000 7f080759e700 1 -- 172.21.15.178:0/1565901438 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08080f62a0 msgr2=0x7f08080f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:08.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.479+0000 7f080759e700 1 --2- 172.21.15.178:0/1565901438 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08080f62a0 0x7f08080f66c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f07f0009fb0 tx=0x7f07f0009b30).stop 2023-11-15T04:56:08.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.480+0000 7f080759e700 1 -- 172.21.15.178:0/1565901438 shutdown_connections 2023-11-15T04:56:08.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.480+0000 7f080759e700 1 --2- 172.21.15.178:0/1565901438 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08080f62a0 0x7f08080f66c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:08.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.480+0000 7f080759e700 1 -- 172.21.15.178:0/1565901438 >> 172.21.15.178:0/1565901438 conn(0x7f08080f1aa0 msgr2=0x7f08080f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:08.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.480+0000 7f080759e700 1 -- 172.21.15.178:0/1565901438 shutdown_connections 2023-11-15T04:56:08.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.480+0000 7f080759e700 1 -- 172.21.15.178:0/1565901438 wait complete. 2023-11-15T04:56:08.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.480+0000 7f080759e700 1 Processor -- start 2023-11-15T04:56:08.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.480+0000 7f080759e700 1 -- start start 2023-11-15T04:56:08.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.480+0000 7f080759e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08080f62a0 0x7f08081064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:08.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.480+0000 7f080759e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08081069a0 con 0x7f08080f62a0 2023-11-15T04:56:08.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.480+0000 7f080659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08080f62a0 0x7f08081064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:08.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.480+0000 7f080659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08080f62a0 0x7f08081064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:48968/0 (socket says 172.21.15.178:48968) 2023-11-15T04:56:08.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.481+0000 7f080659c700 1 -- 172.21.15.178:0/3785555529 learned_addr learned my addr 172.21.15.178:0/3785555529 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:56:08.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.481+0000 7f080659c700 1 -- 172.21.15.178:0/3785555529 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f0005040 con 0x7f08080f62a0 2023-11-15T04:56:08.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.481+0000 7f080659c700 1 --2- 172.21.15.178:0/3785555529 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08080f62a0 0x7f08081064b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f07f0009830 tx=0x7f07f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:08.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.481+0000 7f07feffd700 1 -- 172.21.15.178:0/3785555529 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07f0009320 con 0x7f08080f62a0 2023-11-15T04:56:08.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.481+0000 7f07feffd700 1 -- 172.21.15.178:0/3785555529 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07f00136c0 con 0x7f08080f62a0 2023-11-15T04:56:08.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.481+0000 7f07feffd700 1 -- 172.21.15.178:0/3785555529 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07f001ee90 con 0x7f08080f62a0 2023-11-15T04:56:08.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.481+0000 7f080759e700 1 -- 172.21.15.178:0/3785555529 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0808106ba0 con 0x7f08080f62a0 2023-11-15T04:56:08.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.481+0000 7f080759e700 1 -- 172.21.15.178:0/3785555529 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0808107030 con 0x7f08080f62a0 2023-11-15T04:56:08.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.482+0000 7f07feffd700 1 -- 172.21.15.178:0/3785555529 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07f001e420 con 0x7f08080f62a0 2023-11-15T04:56:08.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.482+0000 7f080759e700 1 -- 172.21.15.178:0/3785555529 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0808045ad0 con 0x7f08080f62a0 2023-11-15T04:56:08.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.482+0000 7f07feffd700 1 --2- 172.21.15.178:0/3785555529 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f07f402ff10 0x7f07f40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:08.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.482+0000 7f07feffd700 1 -- 172.21.15.178:0/3785555529 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f07f0040f40 con 0x7f08080f62a0 2023-11-15T04:56:08.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.482+0000 7f0805d9b700 1 --2- 172.21.15.178:0/3785555529 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f07f402ff10 0x7f07f40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:08.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.483+0000 7f0805d9b700 1 --2- 172.21.15.178:0/3785555529 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f07f402ff10 0x7f07f40323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f07f80096a0 tx=0x7f07f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:08.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.487+0000 7f07feffd700 1 -- 172.21.15.178:0/3785555529 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07f006d2b0 con 0x7f08080f62a0 2023-11-15T04:56:08.694 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.694+0000 7f080759e700 1 -- 172.21.15.178:0/3785555529 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0808107ee0 con 0x7f08080f62a0 2023-11-15T04:56:08.695 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.695+0000 7f07feffd700 1 -- 172.21.15.178:0/3785555529 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f07f00095f0 con 0x7f08080f62a0 2023-11-15T04:56:08.695 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:56:08.696 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:56:08.697 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.696+0000 7f080759e700 1 -- 172.21.15.178:0/3785555529 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f07f402ff10 msgr2=0x7f07f40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:08.697 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.696+0000 7f080759e700 1 --2- 172.21.15.178:0/3785555529 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f07f402ff10 0x7f07f40323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f07f80096a0 tx=0x7f07f8006b40).stop 2023-11-15T04:56:08.697 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.696+0000 7f080759e700 1 -- 172.21.15.178:0/3785555529 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08080f62a0 msgr2=0x7f08081064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:08.697 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.696+0000 7f080759e700 1 --2- 172.21.15.178:0/3785555529 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08080f62a0 0x7f08081064b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f07f0009830 tx=0x7f07f000b040).stop 2023-11-15T04:56:08.698 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.696+0000 7f080759e700 1 -- 172.21.15.178:0/3785555529 shutdown_connections 2023-11-15T04:56:08.698 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.696+0000 7f080759e700 1 --2- 172.21.15.178:0/3785555529 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f07f402ff10 0x7f07f40323d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:08.698 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.697+0000 7f080759e700 1 --2- 172.21.15.178:0/3785555529 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08080f62a0 0x7f08081064b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:08.698 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.697+0000 7f080759e700 1 -- 172.21.15.178:0/3785555529 >> 172.21.15.178:0/3785555529 conn(0x7f08080f1aa0 msgr2=0x7f08080f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:08.698 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.697+0000 7f080759e700 1 -- 172.21.15.178:0/3785555529 shutdown_connections 2023-11-15T04:56:08.699 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:08.697+0000 7f080759e700 1 -- 172.21.15.178:0/3785555529 wait complete. 2023-11-15T04:56:08.699 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:56:09.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:08 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:08.695560+0000 mon.smithi163 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.178:0/3785555529' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:56:10.074 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:56:10.074 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:56:10.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:09 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:08.314209+0000 mgr.smithi163.qesyrx (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:11.273 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.271+0000 7ff0c02ae700 1 -- 172.21.15.178:0/1200063269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0b80f6510 msgr2=0x7ff0b80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:11.274 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.271+0000 7ff0c02ae700 1 --2- 172.21.15.178:0/1200063269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0b80f6510 0x7ff0b80f6930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7ff0b4009fb0 tx=0x7ff0b4009b30).stop 2023-11-15T04:56:11.274 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.271+0000 7ff0c02ae700 1 -- 172.21.15.178:0/1200063269 shutdown_connections 2023-11-15T04:56:11.274 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.271+0000 7ff0c02ae700 1 --2- 172.21.15.178:0/1200063269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0b80f6510 0x7ff0b80f6930 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:11.274 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.271+0000 7ff0c02ae700 1 -- 172.21.15.178:0/1200063269 >> 172.21.15.178:0/1200063269 conn(0x7ff0b80f1c70 msgr2=0x7ff0b80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:11.275 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.271+0000 7ff0c02ae700 1 -- 172.21.15.178:0/1200063269 shutdown_connections 2023-11-15T04:56:11.275 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.271+0000 7ff0c02ae700 1 -- 172.21.15.178:0/1200063269 wait complete. 2023-11-15T04:56:11.275 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.272+0000 7ff0c02ae700 1 Processor -- start 2023-11-15T04:56:11.275 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.272+0000 7ff0c02ae700 1 -- start start 2023-11-15T04:56:11.276 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.272+0000 7ff0c02ae700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0b80f6510 0x7ff0b8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:11.276 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.272+0000 7ff0c02ae700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b8106c20 con 0x7ff0b80f6510 2023-11-15T04:56:11.276 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.272+0000 7ff0be04a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0b80f6510 0x7ff0b8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:11.276 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.272+0000 7ff0be04a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0b80f6510 0x7ff0b8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:48994/0 (socket says 172.21.15.178:48994) 2023-11-15T04:56:11.276 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.272+0000 7ff0be04a700 1 -- 172.21.15.178:0/3758186069 learned_addr learned my addr 172.21.15.178:0/3758186069 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:56:11.277 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.273+0000 7ff0be04a700 1 -- 172.21.15.178:0/3758186069 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b4005040 con 0x7ff0b80f6510 2023-11-15T04:56:11.277 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.273+0000 7ff0be04a700 1 --2- 172.21.15.178:0/3758186069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0b80f6510 0x7ff0b8106730 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7ff0b4013510 tx=0x7ff0b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:11.277 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.273+0000 7ff0aeffd700 1 -- 172.21.15.178:0/3758186069 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0b4009320 con 0x7ff0b80f6510 2023-11-15T04:56:11.278 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.273+0000 7ff0c02ae700 1 -- 172.21.15.178:0/3758186069 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b8106e20 con 0x7ff0b80f6510 2023-11-15T04:56:11.278 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.273+0000 7ff0aeffd700 1 -- 172.21.15.178:0/3758186069 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff0b4013f00 con 0x7ff0b80f6510 2023-11-15T04:56:11.278 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.273+0000 7ff0aeffd700 1 -- 172.21.15.178:0/3758186069 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0b4014d90 con 0x7ff0b80f6510 2023-11-15T04:56:11.279 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.273+0000 7ff0c02ae700 1 -- 172.21.15.178:0/3758186069 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b81072b0 con 0x7ff0b80f6510 2023-11-15T04:56:11.281 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.274+0000 7ff0aeffd700 1 -- 172.21.15.178:0/3758186069 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff0b40263a0 con 0x7ff0b80f6510 2023-11-15T04:56:11.281 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.274+0000 7ff0c02ae700 1 -- 172.21.15.178:0/3758186069 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b8045ad0 con 0x7ff0b80f6510 2023-11-15T04:56:11.281 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.274+0000 7ff0aeffd700 1 --2- 172.21.15.178:0/3758186069 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff0a402ff60 0x7ff0a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:11.282 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.274+0000 7ff0aeffd700 1 -- 172.21.15.178:0/3758186069 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff0b4041010 con 0x7ff0b80f6510 2023-11-15T04:56:11.282 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.274+0000 7ff0bd849700 1 --2- 172.21.15.178:0/3758186069 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff0a402ff60 0x7ff0a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:11.282 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.275+0000 7ff0bd849700 1 --2- 172.21.15.178:0/3758186069 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff0a402ff60 0x7ff0a4032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7ff0a80096a0 tx=0x7ff0a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:11.282 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.279+0000 7ff0aeffd700 1 -- 172.21.15.178:0/3758186069 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff0b40126f0 con 0x7ff0b80f6510 2023-11-15T04:56:11.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.481+0000 7ff0c02ae700 1 -- 172.21.15.178:0/3758186069 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff0b80fbf60 con 0x7ff0b80f6510 2023-11-15T04:56:11.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.482+0000 7ff0aeffd700 1 -- 172.21.15.178:0/3758186069 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff0b4012330 con 0x7ff0b80f6510 2023-11-15T04:56:11.483 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:56:11.483 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:56:11.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.484+0000 7ff0c02ae700 1 -- 172.21.15.178:0/3758186069 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff0a402ff60 msgr2=0x7ff0a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:11.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.484+0000 7ff0c02ae700 1 --2- 172.21.15.178:0/3758186069 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff0a402ff60 0x7ff0a4032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7ff0a80096a0 tx=0x7ff0a8006b40).stop 2023-11-15T04:56:11.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.484+0000 7ff0c02ae700 1 -- 172.21.15.178:0/3758186069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0b80f6510 msgr2=0x7ff0b8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:11.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.484+0000 7ff0c02ae700 1 --2- 172.21.15.178:0/3758186069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0b80f6510 0x7ff0b8106730 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7ff0b4013510 tx=0x7ff0b400b040).stop 2023-11-15T04:56:11.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.484+0000 7ff0c02ae700 1 -- 172.21.15.178:0/3758186069 shutdown_connections 2023-11-15T04:56:11.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.484+0000 7ff0c02ae700 1 --2- 172.21.15.178:0/3758186069 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff0a402ff60 0x7ff0a4032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:11.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.484+0000 7ff0c02ae700 1 --2- 172.21.15.178:0/3758186069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0b80f6510 0x7ff0b8106730 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:11.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.484+0000 7ff0c02ae700 1 -- 172.21.15.178:0/3758186069 >> 172.21.15.178:0/3758186069 conn(0x7ff0b80f1c70 msgr2=0x7ff0b80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:11.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.484+0000 7ff0c02ae700 1 -- 172.21.15.178:0/3758186069 shutdown_connections 2023-11-15T04:56:11.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:11.484+0000 7ff0c02ae700 1 -- 172.21.15.178:0/3758186069 wait complete. 2023-11-15T04:56:11.489 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:56:12.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:11 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:10.314485+0000 mgr.smithi163.qesyrx (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:12.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:11 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:11.482792+0000 mon.smithi163 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.178:0/3758186069' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:56:12.863 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:56:12.865 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:56:14.046 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.043+0000 7efe34fa0700 1 -- 172.21.15.178:0/2317274347 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe300f62a0 msgr2=0x7efe300f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:14.046 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.043+0000 7efe34fa0700 1 --2- 172.21.15.178:0/2317274347 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe300f62a0 0x7efe300f66c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7efe24009fb0 tx=0x7efe24009b30).stop 2023-11-15T04:56:14.046 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.044+0000 7efe34fa0700 1 -- 172.21.15.178:0/2317274347 shutdown_connections 2023-11-15T04:56:14.047 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.044+0000 7efe34fa0700 1 --2- 172.21.15.178:0/2317274347 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe300f62a0 0x7efe300f66c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:14.047 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.044+0000 7efe34fa0700 1 -- 172.21.15.178:0/2317274347 >> 172.21.15.178:0/2317274347 conn(0x7efe300f1aa0 msgr2=0x7efe300f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:14.047 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.044+0000 7efe34fa0700 1 -- 172.21.15.178:0/2317274347 shutdown_connections 2023-11-15T04:56:14.047 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.044+0000 7efe34fa0700 1 -- 172.21.15.178:0/2317274347 wait complete. 2023-11-15T04:56:14.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.044+0000 7efe34fa0700 1 Processor -- start 2023-11-15T04:56:14.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.044+0000 7efe34fa0700 1 -- start start 2023-11-15T04:56:14.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.044+0000 7efe34fa0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe300f62a0 0x7efe301064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:14.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.044+0000 7efe34fa0700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe301069a0 con 0x7efe300f62a0 2023-11-15T04:56:14.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.045+0000 7efe2f7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe300f62a0 0x7efe301064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:14.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.045+0000 7efe2f7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe300f62a0 0x7efe301064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:49010/0 (socket says 172.21.15.178:49010) 2023-11-15T04:56:14.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.045+0000 7efe2f7fe700 1 -- 172.21.15.178:0/2722067999 learned_addr learned my addr 172.21.15.178:0/2722067999 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:56:14.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.045+0000 7efe2f7fe700 1 -- 172.21.15.178:0/2722067999 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe24005040 con 0x7efe300f62a0 2023-11-15T04:56:14.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.045+0000 7efe2f7fe700 1 --2- 172.21.15.178:0/2722067999 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe300f62a0 0x7efe301064b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7efe24013510 tx=0x7efe2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:14.050 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.045+0000 7efe17fff700 1 -- 172.21.15.178:0/2722067999 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe24009320 con 0x7efe300f62a0 2023-11-15T04:56:14.050 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.045+0000 7efe17fff700 1 -- 172.21.15.178:0/2722067999 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe24013f00 con 0x7efe300f62a0 2023-11-15T04:56:14.050 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.045+0000 7efe17fff700 1 -- 172.21.15.178:0/2722067999 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe24014dc0 con 0x7efe300f62a0 2023-11-15T04:56:14.050 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.045+0000 7efe34fa0700 1 -- 172.21.15.178:0/2722067999 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe30106ba0 con 0x7efe300f62a0 2023-11-15T04:56:14.050 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.045+0000 7efe34fa0700 1 -- 172.21.15.178:0/2722067999 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe30107030 con 0x7efe300f62a0 2023-11-15T04:56:14.052 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.046+0000 7efe17fff700 1 -- 172.21.15.178:0/2722067999 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe240095f0 con 0x7efe300f62a0 2023-11-15T04:56:14.052 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.046+0000 7efe34fa0700 1 -- 172.21.15.178:0/2722067999 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe30045ad0 con 0x7efe300f62a0 2023-11-15T04:56:14.052 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.046+0000 7efe17fff700 1 --2- 172.21.15.178:0/2722067999 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7efe1802ffb0 0x7efe18032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:14.052 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.046+0000 7efe17fff700 1 -- 172.21.15.178:0/2722067999 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe240410c0 con 0x7efe300f62a0 2023-11-15T04:56:14.052 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.046+0000 7efe2effd700 1 --2- 172.21.15.178:0/2722067999 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7efe1802ffb0 0x7efe18032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:14.053 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.047+0000 7efe2effd700 1 --2- 172.21.15.178:0/2722067999 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7efe1802ffb0 0x7efe18032470 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7efe200096a0 tx=0x7efe20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:14.053 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.051+0000 7efe17fff700 1 -- 172.21.15.178:0/2722067999 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe24014420 con 0x7efe300f62a0 2023-11-15T04:56:14.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:13 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:12.314741+0000 mgr.smithi163.qesyrx (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:14.256 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.256+0000 7efe34fa0700 1 -- 172.21.15.178:0/2722067999 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe30107ee0 con 0x7efe300f62a0 2023-11-15T04:56:14.257 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.256+0000 7efe17fff700 1 -- 172.21.15.178:0/2722067999 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efe24010230 con 0x7efe300f62a0 2023-11-15T04:56:14.257 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:56:14.258 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:56:14.259 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.258+0000 7efe34fa0700 1 -- 172.21.15.178:0/2722067999 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7efe1802ffb0 msgr2=0x7efe18032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:14.259 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.258+0000 7efe34fa0700 1 --2- 172.21.15.178:0/2722067999 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7efe1802ffb0 0x7efe18032470 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7efe200096a0 tx=0x7efe20006b40).stop 2023-11-15T04:56:14.259 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.258+0000 7efe34fa0700 1 -- 172.21.15.178:0/2722067999 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe300f62a0 msgr2=0x7efe301064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:14.259 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.258+0000 7efe34fa0700 1 --2- 172.21.15.178:0/2722067999 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe300f62a0 0x7efe301064b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7efe24013510 tx=0x7efe2400b040).stop 2023-11-15T04:56:14.259 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.258+0000 7efe34fa0700 1 -- 172.21.15.178:0/2722067999 shutdown_connections 2023-11-15T04:56:14.260 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.258+0000 7efe34fa0700 1 --2- 172.21.15.178:0/2722067999 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7efe1802ffb0 0x7efe18032470 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:14.260 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.258+0000 7efe34fa0700 1 --2- 172.21.15.178:0/2722067999 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe300f62a0 0x7efe301064b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:14.260 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.258+0000 7efe34fa0700 1 -- 172.21.15.178:0/2722067999 >> 172.21.15.178:0/2722067999 conn(0x7efe300f1aa0 msgr2=0x7efe300f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:14.260 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.258+0000 7efe34fa0700 1 -- 172.21.15.178:0/2722067999 shutdown_connections 2023-11-15T04:56:14.260 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:14.258+0000 7efe34fa0700 1 -- 172.21.15.178:0/2722067999 wait complete. 2023-11-15T04:56:14.261 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:56:15.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:14 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:14.257261+0000 mon.smithi163 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.178:0/2722067999' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:56:15.633 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:56:15.634 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:56:16.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:15 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:14.315045+0000 mgr.smithi163.qesyrx (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:16.744 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.742+0000 7fec987be700 1 -- 172.21.15.178:0/3586870522 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec900f6510 msgr2=0x7fec900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:16.744 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.742+0000 7fec987be700 1 --2- 172.21.15.178:0/3586870522 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec900f6510 0x7fec900f6930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fec84009fb0 tx=0x7fec84009b30).stop 2023-11-15T04:56:16.745 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.742+0000 7fec987be700 1 -- 172.21.15.178:0/3586870522 shutdown_connections 2023-11-15T04:56:16.745 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.742+0000 7fec987be700 1 --2- 172.21.15.178:0/3586870522 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec900f6510 0x7fec900f6930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:16.745 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.742+0000 7fec987be700 1 -- 172.21.15.178:0/3586870522 >> 172.21.15.178:0/3586870522 conn(0x7fec900f1c70 msgr2=0x7fec900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:16.745 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.742+0000 7fec987be700 1 -- 172.21.15.178:0/3586870522 shutdown_connections 2023-11-15T04:56:16.745 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.742+0000 7fec987be700 1 -- 172.21.15.178:0/3586870522 wait complete. 2023-11-15T04:56:16.746 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.743+0000 7fec987be700 1 Processor -- start 2023-11-15T04:56:16.746 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.743+0000 7fec987be700 1 -- start start 2023-11-15T04:56:16.746 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.743+0000 7fec987be700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec900f6510 0x7fec90106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:16.746 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.743+0000 7fec987be700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec90106c20 con 0x7fec900f6510 2023-11-15T04:56:16.746 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.743+0000 7fec9655a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec900f6510 0x7fec90106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:16.747 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.743+0000 7fec9655a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec900f6510 0x7fec90106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:59634/0 (socket says 172.21.15.178:59634) 2023-11-15T04:56:16.747 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.743+0000 7fec9655a700 1 -- 172.21.15.178:0/865080382 learned_addr learned my addr 172.21.15.178:0/865080382 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:56:16.747 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.743+0000 7fec9655a700 1 -- 172.21.15.178:0/865080382 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec84005040 con 0x7fec900f6510 2023-11-15T04:56:16.747 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.743+0000 7fec9655a700 1 --2- 172.21.15.178:0/865080382 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec900f6510 0x7fec90106730 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fec84013510 tx=0x7fec8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:16.747 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.744+0000 7fec82ffd700 1 -- 172.21.15.178:0/865080382 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec84009320 con 0x7fec900f6510 2023-11-15T04:56:16.748 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.744+0000 7fec82ffd700 1 -- 172.21.15.178:0/865080382 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec84013f00 con 0x7fec900f6510 2023-11-15T04:56:16.748 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.744+0000 7fec987be700 1 -- 172.21.15.178:0/865080382 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec90106e20 con 0x7fec900f6510 2023-11-15T04:56:16.748 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.744+0000 7fec82ffd700 1 -- 172.21.15.178:0/865080382 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec84014d90 con 0x7fec900f6510 2023-11-15T04:56:16.748 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.744+0000 7fec987be700 1 -- 172.21.15.178:0/865080382 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec901072b0 con 0x7fec900f6510 2023-11-15T04:56:16.751 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.744+0000 7fec82ffd700 1 -- 172.21.15.178:0/865080382 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec840095f0 con 0x7fec900f6510 2023-11-15T04:56:16.751 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.744+0000 7fec987be700 1 -- 172.21.15.178:0/865080382 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec90045ad0 con 0x7fec900f6510 2023-11-15T04:56:16.752 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.745+0000 7fec82ffd700 1 --2- 172.21.15.178:0/865080382 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fec7c02ff60 0x7fec7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:16.752 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.745+0000 7fec82ffd700 1 -- 172.21.15.178:0/865080382 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec84040fe0 con 0x7fec900f6510 2023-11-15T04:56:16.752 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.745+0000 7fec95d59700 1 --2- 172.21.15.178:0/865080382 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fec7c02ff60 0x7fec7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:16.752 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.745+0000 7fec95d59700 1 --2- 172.21.15.178:0/865080382 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fec7c02ff60 0x7fec7c032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fec8c0096a0 tx=0x7fec8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:16.752 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.749+0000 7fec82ffd700 1 -- 172.21.15.178:0/865080382 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec8401d7f0 con 0x7fec900f6510 2023-11-15T04:56:16.955 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.954+0000 7fec987be700 1 -- 172.21.15.178:0/865080382 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec900fbf60 con 0x7fec900f6510 2023-11-15T04:56:16.956 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.955+0000 7fec82ffd700 1 -- 172.21.15.178:0/865080382 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fec84010230 con 0x7fec900f6510 2023-11-15T04:56:16.956 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:56:16.957 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:56:16.958 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.956+0000 7fec987be700 1 -- 172.21.15.178:0/865080382 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fec7c02ff60 msgr2=0x7fec7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:16.958 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.956+0000 7fec987be700 1 --2- 172.21.15.178:0/865080382 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fec7c02ff60 0x7fec7c032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fec8c0096a0 tx=0x7fec8c006b40).stop 2023-11-15T04:56:16.958 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.957+0000 7fec987be700 1 -- 172.21.15.178:0/865080382 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec900f6510 msgr2=0x7fec90106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:16.959 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.957+0000 7fec987be700 1 --2- 172.21.15.178:0/865080382 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec900f6510 0x7fec90106730 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fec84013510 tx=0x7fec8400b040).stop 2023-11-15T04:56:16.959 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.957+0000 7fec987be700 1 -- 172.21.15.178:0/865080382 shutdown_connections 2023-11-15T04:56:16.960 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.957+0000 7fec987be700 1 --2- 172.21.15.178:0/865080382 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fec7c02ff60 0x7fec7c032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:16.960 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.957+0000 7fec987be700 1 --2- 172.21.15.178:0/865080382 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec900f6510 0x7fec90106730 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:16.960 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.957+0000 7fec987be700 1 -- 172.21.15.178:0/865080382 >> 172.21.15.178:0/865080382 conn(0x7fec900f1c70 msgr2=0x7fec900f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:16.961 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.957+0000 7fec987be700 1 -- 172.21.15.178:0/865080382 shutdown_connections 2023-11-15T04:56:16.962 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:16.957+0000 7fec987be700 1 -- 172.21.15.178:0/865080382 wait complete. 2023-11-15T04:56:16.962 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:56:18.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:17 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:16.315353+0000 mgr.smithi163.qesyrx (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:18.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:17 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:16.955457+0000 mon.smithi163 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.178:0/865080382' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:56:18.337 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:56:18.338 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:56:20.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:19 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:18.315718+0000 mgr.smithi163.qesyrx (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:22.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:21 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:20.315980+0000 mgr.smithi163.qesyrx (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:23.987 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.918+0000 7f9ec774e700 1 -- 172.21.15.178:0/1092923804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ec0060200 msgr2=0x7f9ec00f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:23.988 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.918+0000 7f9ec774e700 1 --2- 172.21.15.178:0/1092923804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ec0060200 0x7f9ec00f9170 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f9eb0009fb0 tx=0x7f9eb0009b30).stop 2023-11-15T04:56:23.988 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.918+0000 7f9ec774e700 1 -- 172.21.15.178:0/1092923804 shutdown_connections 2023-11-15T04:56:23.988 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.918+0000 7f9ec774e700 1 --2- 172.21.15.178:0/1092923804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ec0060200 0x7f9ec00f9170 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:23.988 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.918+0000 7f9ec774e700 1 -- 172.21.15.178:0/1092923804 >> 172.21.15.178:0/1092923804 conn(0x7f9ec00f0b70 msgr2=0x7f9ec00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:23.988 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.918+0000 7f9ec774e700 1 -- 172.21.15.178:0/1092923804 shutdown_connections 2023-11-15T04:56:23.989 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.918+0000 7f9ec774e700 1 -- 172.21.15.178:0/1092923804 wait complete. 2023-11-15T04:56:23.989 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.918+0000 7f9ec774e700 1 Processor -- start 2023-11-15T04:56:23.989 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.918+0000 7f9ec774e700 1 -- start start 2023-11-15T04:56:23.989 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.919+0000 7f9ec774e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ec0060200 0x7f9ec01066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:23.989 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.919+0000 7f9ec774e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec0106b90 con 0x7f9ec0060200 2023-11-15T04:56:23.990 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.919+0000 7f9ec54ea700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ec0060200 0x7f9ec01066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:23.990 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.919+0000 7f9ec54ea700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ec0060200 0x7f9ec01066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:59652/0 (socket says 172.21.15.178:59652) 2023-11-15T04:56:23.990 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.919+0000 7f9ec54ea700 1 -- 172.21.15.178:0/3317498669 learned_addr learned my addr 172.21.15.178:0/3317498669 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:56:23.990 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.919+0000 7f9ec54ea700 1 -- 172.21.15.178:0/3317498669 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb0005040 con 0x7f9ec0060200 2023-11-15T04:56:23.990 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.919+0000 7f9ec54ea700 1 --2- 172.21.15.178:0/3317498669 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ec0060200 0x7f9ec01066a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f9eb0013510 tx=0x7f9eb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:23.991 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.919+0000 7f9eb5ffb700 1 -- 172.21.15.178:0/3317498669 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9eb0009320 con 0x7f9ec0060200 2023-11-15T04:56:23.991 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.920+0000 7f9eb5ffb700 1 -- 172.21.15.178:0/3317498669 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9eb0013f00 con 0x7f9ec0060200 2023-11-15T04:56:23.991 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.920+0000 7f9ec774e700 1 -- 172.21.15.178:0/3317498669 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec0106d90 con 0x7f9ec0060200 2023-11-15T04:56:23.991 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.920+0000 7f9eb5ffb700 1 -- 172.21.15.178:0/3317498669 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9eb0014dc0 con 0x7f9ec0060200 2023-11-15T04:56:23.991 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.920+0000 7f9ec774e700 1 -- 172.21.15.178:0/3317498669 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec0107220 con 0x7f9ec0060200 2023-11-15T04:56:23.992 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.921+0000 7f9eb5ffb700 1 -- 172.21.15.178:0/3317498669 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9eb00095f0 con 0x7f9ec0060200 2023-11-15T04:56:23.992 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.921+0000 7f9ec774e700 1 -- 172.21.15.178:0/3317498669 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec0045ad0 con 0x7f9ec0060200 2023-11-15T04:56:23.992 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.921+0000 7f9eb5ffb700 1 --2- 172.21.15.178:0/3317498669 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f9eac02ffb0 0x7f9eac032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:23.992 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.921+0000 7f9eb5ffb700 1 -- 172.21.15.178:0/3317498669 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9eb0040f90 con 0x7f9ec0060200 2023-11-15T04:56:23.992 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.921+0000 7f9ec4ce9700 1 --2- 172.21.15.178:0/3317498669 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f9eac02ffb0 0x7f9eac032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:23.993 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.922+0000 7f9ec4ce9700 1 --2- 172.21.15.178:0/3317498669 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f9eac02ffb0 0x7f9eac032470 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f9ebc0096a0 tx=0x7f9ebc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:23.993 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:23.926+0000 7f9eb5ffb700 1 -- 172.21.15.178:0/3317498669 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9eb0014420 con 0x7f9ec0060200 2023-11-15T04:56:24.135 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:24.134+0000 7f9ec774e700 1 -- 172.21.15.178:0/3317498669 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ec0108070 con 0x7f9ec0060200 2023-11-15T04:56:24.137 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:24.135+0000 7f9eb5ffb700 1 -- 172.21.15.178:0/3317498669 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9eb0010230 con 0x7f9ec0060200 2023-11-15T04:56:24.138 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:56:24.138 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:56:24.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:24.136+0000 7f9ec774e700 1 -- 172.21.15.178:0/3317498669 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f9eac02ffb0 msgr2=0x7f9eac032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:24.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:24.136+0000 7f9ec774e700 1 --2- 172.21.15.178:0/3317498669 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f9eac02ffb0 0x7f9eac032470 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f9ebc0096a0 tx=0x7f9ebc006b40).stop 2023-11-15T04:56:24.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:24.136+0000 7f9ec774e700 1 -- 172.21.15.178:0/3317498669 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ec0060200 msgr2=0x7f9ec01066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:24.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:24.136+0000 7f9ec774e700 1 --2- 172.21.15.178:0/3317498669 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ec0060200 0x7f9ec01066a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f9eb0013510 tx=0x7f9eb000b040).stop 2023-11-15T04:56:24.140 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:24.136+0000 7f9ec774e700 1 -- 172.21.15.178:0/3317498669 shutdown_connections 2023-11-15T04:56:24.140 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:24.136+0000 7f9ec774e700 1 --2- 172.21.15.178:0/3317498669 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f9eac02ffb0 0x7f9eac032470 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:24.140 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:24.136+0000 7f9ec774e700 1 --2- 172.21.15.178:0/3317498669 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ec0060200 0x7f9ec01066a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:24.140 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:24.136+0000 7f9ec774e700 1 -- 172.21.15.178:0/3317498669 >> 172.21.15.178:0/3317498669 conn(0x7f9ec00f0b70 msgr2=0x7f9ec00f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:24.140 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:24.136+0000 7f9ec774e700 1 -- 172.21.15.178:0/3317498669 shutdown_connections 2023-11-15T04:56:24.141 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:24.136+0000 7f9ec774e700 1 -- 172.21.15.178:0/3317498669 wait complete. 2023-11-15T04:56:24.141 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:56:24.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:23 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:22.316339+0000 mgr.smithi163.qesyrx (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:25.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:24 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:24.136354+0000 mon.smithi163 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.178:0/3317498669' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:56:25.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:24 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:24.448361+0000 mon.smithi163 (mon.0) 128 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:56:25.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:24 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:24.450407+0000 mon.smithi163 (mon.0) 129 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/trash_purge_schedule"}]: dispatch 2023-11-15T04:56:26.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:24.316672+0000 mgr.smithi163.qesyrx (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:27.164 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:56:27.165 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:56:28.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:27 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:26.317099+0000 mgr.smithi163.qesyrx (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:28.359 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.357+0000 7fe17b598700 1 -- 172.21.15.178:0/1207293861 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1740f47c0 msgr2=0x7fe1740f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:28.360 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.357+0000 7fe17b598700 1 --2- 172.21.15.178:0/1207293861 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1740f47c0 0x7fe1740f4be0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fe168009fb0 tx=0x7fe168009b30).stop 2023-11-15T04:56:28.360 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.357+0000 7fe17b598700 1 -- 172.21.15.178:0/1207293861 shutdown_connections 2023-11-15T04:56:28.360 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.357+0000 7fe17b598700 1 --2- 172.21.15.178:0/1207293861 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1740f47c0 0x7fe1740f4be0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:28.360 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.357+0000 7fe17b598700 1 -- 172.21.15.178:0/1207293861 >> 172.21.15.178:0/1207293861 conn(0x7fe1740f02e0 msgr2=0x7fe1740f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:28.360 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.357+0000 7fe17b598700 1 -- 172.21.15.178:0/1207293861 shutdown_connections 2023-11-15T04:56:28.361 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.357+0000 7fe17b598700 1 -- 172.21.15.178:0/1207293861 wait complete. 2023-11-15T04:56:28.361 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.358+0000 7fe17b598700 1 Processor -- start 2023-11-15T04:56:28.361 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.358+0000 7fe17b598700 1 -- start start 2023-11-15T04:56:28.361 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.358+0000 7fe17b598700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1740f47c0 0x7fe174106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:28.361 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.358+0000 7fe17b598700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe174106bf0 con 0x7fe1740f47c0 2023-11-15T04:56:28.362 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.358+0000 7fe179334700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1740f47c0 0x7fe174106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:28.362 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.358+0000 7fe179334700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1740f47c0 0x7fe174106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:51242/0 (socket says 172.21.15.178:51242) 2023-11-15T04:56:28.362 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.358+0000 7fe179334700 1 -- 172.21.15.178:0/577538760 learned_addr learned my addr 172.21.15.178:0/577538760 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:56:28.363 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.358+0000 7fe179334700 1 -- 172.21.15.178:0/577538760 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe168005040 con 0x7fe1740f47c0 2023-11-15T04:56:28.364 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.358+0000 7fe179334700 1 --2- 172.21.15.178:0/577538760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1740f47c0 0x7fe174106700 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fe168013510 tx=0x7fe16800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:28.364 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.359+0000 7fe165ffb700 1 -- 172.21.15.178:0/577538760 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe168009320 con 0x7fe1740f47c0 2023-11-15T04:56:28.364 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.359+0000 7fe17b598700 1 -- 172.21.15.178:0/577538760 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe174106df0 con 0x7fe1740f47c0 2023-11-15T04:56:28.364 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.359+0000 7fe165ffb700 1 -- 172.21.15.178:0/577538760 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe168013f00 con 0x7fe1740f47c0 2023-11-15T04:56:28.364 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.359+0000 7fe165ffb700 1 -- 172.21.15.178:0/577538760 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe168014d90 con 0x7fe1740f47c0 2023-11-15T04:56:28.365 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.359+0000 7fe17b598700 1 -- 172.21.15.178:0/577538760 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe174107280 con 0x7fe1740f47c0 2023-11-15T04:56:28.365 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.360+0000 7fe165ffb700 1 -- 172.21.15.178:0/577538760 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe168014420 con 0x7fe1740f47c0 2023-11-15T04:56:28.365 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.360+0000 7fe17b598700 1 -- 172.21.15.178:0/577538760 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe174045ad0 con 0x7fe1740f47c0 2023-11-15T04:56:28.365 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.360+0000 7fe165ffb700 1 --2- 172.21.15.178:0/577538760 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe16002ff60 0x7fe160032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:28.365 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.360+0000 7fe165ffb700 1 -- 172.21.15.178:0/577538760 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe168041050 con 0x7fe1740f47c0 2023-11-15T04:56:28.366 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.360+0000 7fe178b33700 1 --2- 172.21.15.178:0/577538760 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe16002ff60 0x7fe160032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:28.366 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.361+0000 7fe178b33700 1 --2- 172.21.15.178:0/577538760 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe16002ff60 0x7fe160032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fe1700096a0 tx=0x7fe170006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:28.366 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.365+0000 7fe165ffb700 1 -- 172.21.15.178:0/577538760 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe1680095f0 con 0x7fe1740f47c0 2023-11-15T04:56:28.573 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.572+0000 7fe17b598700 1 -- 172.21.15.178:0/577538760 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1741080d0 con 0x7fe1740f47c0 2023-11-15T04:56:28.574 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.572+0000 7fe165ffb700 1 -- 172.21.15.178:0/577538760 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe16801db70 con 0x7fe1740f47c0 2023-11-15T04:56:28.574 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:56:28.575 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:56:28.576 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.574+0000 7fe17b598700 1 -- 172.21.15.178:0/577538760 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe16002ff60 msgr2=0x7fe160032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:28.576 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.574+0000 7fe17b598700 1 --2- 172.21.15.178:0/577538760 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe16002ff60 0x7fe160032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fe1700096a0 tx=0x7fe170006b40).stop 2023-11-15T04:56:28.576 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.574+0000 7fe17b598700 1 -- 172.21.15.178:0/577538760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1740f47c0 msgr2=0x7fe174106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:28.576 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.574+0000 7fe17b598700 1 --2- 172.21.15.178:0/577538760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1740f47c0 0x7fe174106700 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fe168013510 tx=0x7fe16800b040).stop 2023-11-15T04:56:28.577 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.574+0000 7fe17b598700 1 -- 172.21.15.178:0/577538760 shutdown_connections 2023-11-15T04:56:28.577 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.574+0000 7fe17b598700 1 --2- 172.21.15.178:0/577538760 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe16002ff60 0x7fe160032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:28.577 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.574+0000 7fe17b598700 1 --2- 172.21.15.178:0/577538760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1740f47c0 0x7fe174106700 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:28.577 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.574+0000 7fe17b598700 1 -- 172.21.15.178:0/577538760 >> 172.21.15.178:0/577538760 conn(0x7fe1740f02e0 msgr2=0x7fe1740f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:28.577 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.574+0000 7fe17b598700 1 -- 172.21.15.178:0/577538760 shutdown_connections 2023-11-15T04:56:28.578 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:28.574+0000 7fe17b598700 1 -- 172.21.15.178:0/577538760 wait complete. 2023-11-15T04:56:28.578 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:56:29.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:28 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:28.574075+0000 mon.smithi163 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.178:0/577538760' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:56:29.956 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:56:29.957 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:56:30.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:29 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:28.317348+0000 mgr.smithi163.qesyrx (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:31.100 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.098+0000 7f0179f40700 1 -- 172.21.15.178:0/1165505361 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01740f5410 msgr2=0x7f01740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:31.100 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.098+0000 7f0179f40700 1 --2- 172.21.15.178:0/1165505361 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01740f5410 0x7f01740f5830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f015c009fb0 tx=0x7f015c009b30).stop 2023-11-15T04:56:31.100 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.098+0000 7f0179f40700 1 -- 172.21.15.178:0/1165505361 shutdown_connections 2023-11-15T04:56:31.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.098+0000 7f0179f40700 1 --2- 172.21.15.178:0/1165505361 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01740f5410 0x7f01740f5830 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:31.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.098+0000 7f0179f40700 1 -- 172.21.15.178:0/1165505361 >> 172.21.15.178:0/1165505361 conn(0x7f01740f0b70 msgr2=0x7f01740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:31.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.098+0000 7f0179f40700 1 -- 172.21.15.178:0/1165505361 shutdown_connections 2023-11-15T04:56:31.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.098+0000 7f0179f40700 1 -- 172.21.15.178:0/1165505361 wait complete. 2023-11-15T04:56:31.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.098+0000 7f0179f40700 1 Processor -- start 2023-11-15T04:56:31.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.098+0000 7f0179f40700 1 -- start start 2023-11-15T04:56:31.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.099+0000 7f0179f40700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01740f5410 0x7f0174109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:31.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.099+0000 7f0179f40700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0174060220 con 0x7f01740f5410 2023-11-15T04:56:31.104 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.099+0000 7f01737fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01740f5410 0x7f0174109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:31.104 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.099+0000 7f01737fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01740f5410 0x7f0174109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:51258/0 (socket says 172.21.15.178:51258) 2023-11-15T04:56:31.104 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.099+0000 7f01737fe700 1 -- 172.21.15.178:0/86794643 learned_addr learned my addr 172.21.15.178:0/86794643 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:56:31.104 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.099+0000 7f01737fe700 1 -- 172.21.15.178:0/86794643 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c005040 con 0x7f01740f5410 2023-11-15T04:56:31.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.099+0000 7f01737fe700 1 --2- 172.21.15.178:0/86794643 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01740f5410 0x7f0174109450 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f015c013510 tx=0x7f015c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:31.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.100+0000 7f016bfff700 1 -- 172.21.15.178:0/86794643 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f015c009320 con 0x7f01740f5410 2023-11-15T04:56:31.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.100+0000 7f0179f40700 1 -- 172.21.15.178:0/86794643 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0174060420 con 0x7f01740f5410 2023-11-15T04:56:31.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.100+0000 7f016bfff700 1 -- 172.21.15.178:0/86794643 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f015c013f00 con 0x7f01740f5410 2023-11-15T04:56:31.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.100+0000 7f016bfff700 1 -- 172.21.15.178:0/86794643 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f015c014d90 con 0x7f01740f5410 2023-11-15T04:56:31.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.100+0000 7f0179f40700 1 -- 172.21.15.178:0/86794643 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0174109b00 con 0x7f01740f5410 2023-11-15T04:56:31.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.100+0000 7f016bfff700 1 -- 172.21.15.178:0/86794643 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f015c0095f0 con 0x7f01740f5410 2023-11-15T04:56:31.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.101+0000 7f0179f40700 1 -- 172.21.15.178:0/86794643 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0174059100 con 0x7f01740f5410 2023-11-15T04:56:31.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.101+0000 7f016bfff700 1 --2- 172.21.15.178:0/86794643 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f016002ff60 0x7f0160032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:31.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.101+0000 7f016bfff700 1 -- 172.21.15.178:0/86794643 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f015c0410a0 con 0x7f01740f5410 2023-11-15T04:56:31.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.101+0000 7f0172ffd700 1 --2- 172.21.15.178:0/86794643 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f016002ff60 0x7f0160032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:31.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.101+0000 7f0172ffd700 1 --2- 172.21.15.178:0/86794643 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f016002ff60 0x7f0160032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f01640096a0 tx=0x7f0164006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:31.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.106+0000 7f016bfff700 1 -- 172.21.15.178:0/86794643 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f015c01d7f0 con 0x7f01740f5410 2023-11-15T04:56:31.313 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.312+0000 7f0179f40700 1 -- 172.21.15.178:0/86794643 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0174045ad0 con 0x7f01740f5410 2023-11-15T04:56:31.313 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.312+0000 7f016bfff700 1 -- 172.21.15.178:0/86794643 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f015c010230 con 0x7f01740f5410 2023-11-15T04:56:31.314 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:56:31.314 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:56:31.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.314+0000 7f0179f40700 1 -- 172.21.15.178:0/86794643 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f016002ff60 msgr2=0x7f0160032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:31.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.314+0000 7f0179f40700 1 --2- 172.21.15.178:0/86794643 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f016002ff60 0x7f0160032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f01640096a0 tx=0x7f0164006b40).stop 2023-11-15T04:56:31.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.314+0000 7f0179f40700 1 -- 172.21.15.178:0/86794643 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01740f5410 msgr2=0x7f0174109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:31.316 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.314+0000 7f0179f40700 1 --2- 172.21.15.178:0/86794643 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01740f5410 0x7f0174109450 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f015c013510 tx=0x7f015c00b040).stop 2023-11-15T04:56:31.316 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.314+0000 7f0179f40700 1 -- 172.21.15.178:0/86794643 shutdown_connections 2023-11-15T04:56:31.316 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.314+0000 7f0179f40700 1 --2- 172.21.15.178:0/86794643 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f016002ff60 0x7f0160032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:31.316 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.314+0000 7f0179f40700 1 --2- 172.21.15.178:0/86794643 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01740f5410 0x7f0174109450 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:31.316 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.314+0000 7f0179f40700 1 -- 172.21.15.178:0/86794643 >> 172.21.15.178:0/86794643 conn(0x7f01740f0b70 msgr2=0x7f01740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:31.317 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.314+0000 7f0179f40700 1 -- 172.21.15.178:0/86794643 shutdown_connections 2023-11-15T04:56:31.317 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:31.314+0000 7f0179f40700 1 -- 172.21.15.178:0/86794643 wait complete. 2023-11-15T04:56:31.319 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:56:32.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:31 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:30.317568+0000 mgr.smithi163.qesyrx (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:32.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:31 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:31.313844+0000 mon.smithi163 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.178:0/86794643' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:56:32.705 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:56:32.705 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:56:33.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.849+0000 7f5748212700 1 -- 172.21.15.178:0/230516946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57400f4b00 msgr2=0x7f57400f4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:33.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.849+0000 7f5748212700 1 --2- 172.21.15.178:0/230516946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57400f4b00 0x7f57400f4f20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f5730009fb0 tx=0x7f5730009b30).stop 2023-11-15T04:56:33.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.849+0000 7f5748212700 1 -- 172.21.15.178:0/230516946 shutdown_connections 2023-11-15T04:56:33.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.849+0000 7f5748212700 1 --2- 172.21.15.178:0/230516946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57400f4b00 0x7f57400f4f20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:33.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.849+0000 7f5748212700 1 -- 172.21.15.178:0/230516946 >> 172.21.15.178:0/230516946 conn(0x7f57400f0260 msgr2=0x7f57400f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:33.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.849+0000 7f5748212700 1 -- 172.21.15.178:0/230516946 shutdown_connections 2023-11-15T04:56:33.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.849+0000 7f5748212700 1 -- 172.21.15.178:0/230516946 wait complete. 2023-11-15T04:56:33.854 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.850+0000 7f5748212700 1 Processor -- start 2023-11-15T04:56:33.854 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.850+0000 7f5748212700 1 -- start start 2023-11-15T04:56:33.855 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.850+0000 7f5748212700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57400f4b00 0x7f5740106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:33.855 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.850+0000 7f5748212700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5740106c00 con 0x7f57400f4b00 2023-11-15T04:56:33.855 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.850+0000 7f5745fae700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57400f4b00 0x7f5740106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:33.855 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.850+0000 7f5745fae700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57400f4b00 0x7f5740106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:51278/0 (socket says 172.21.15.178:51278) 2023-11-15T04:56:33.855 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.850+0000 7f5745fae700 1 -- 172.21.15.178:0/3693914849 learned_addr learned my addr 172.21.15.178:0/3693914849 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:56:33.856 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.851+0000 7f5745fae700 1 -- 172.21.15.178:0/3693914849 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5730005040 con 0x7f57400f4b00 2023-11-15T04:56:33.857 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.851+0000 7f5745fae700 1 --2- 172.21.15.178:0/3693914849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57400f4b00 0x7f5740106710 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f5730013510 tx=0x7f573000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:33.857 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.851+0000 7f57367fc700 1 -- 172.21.15.178:0/3693914849 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5730009320 con 0x7f57400f4b00 2023-11-15T04:56:33.857 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.851+0000 7f57367fc700 1 -- 172.21.15.178:0/3693914849 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5730013f00 con 0x7f57400f4b00 2023-11-15T04:56:33.857 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.851+0000 7f5748212700 1 -- 172.21.15.178:0/3693914849 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5740106e00 con 0x7f57400f4b00 2023-11-15T04:56:33.858 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.851+0000 7f57367fc700 1 -- 172.21.15.178:0/3693914849 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5730014d90 con 0x7f57400f4b00 2023-11-15T04:56:33.858 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.851+0000 7f5748212700 1 -- 172.21.15.178:0/3693914849 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5740107290 con 0x7f57400f4b00 2023-11-15T04:56:33.858 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.852+0000 7f57367fc700 1 -- 172.21.15.178:0/3693914849 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57300095f0 con 0x7f57400f4b00 2023-11-15T04:56:33.858 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.852+0000 7f5748212700 1 -- 172.21.15.178:0/3693914849 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5740045ad0 con 0x7f57400f4b00 2023-11-15T04:56:33.858 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.852+0000 7f57367fc700 1 --2- 172.21.15.178:0/3693914849 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f572c02ff60 0x7f572c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:33.858 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.852+0000 7f57367fc700 1 -- 172.21.15.178:0/3693914849 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5730041000 con 0x7f57400f4b00 2023-11-15T04:56:33.859 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.852+0000 7f57457ad700 1 --2- 172.21.15.178:0/3693914849 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f572c02ff60 0x7f572c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:33.859 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.853+0000 7f57457ad700 1 --2- 172.21.15.178:0/3693914849 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f572c02ff60 0x7f572c032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f573c0096a0 tx=0x7f573c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:33.860 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:33.857+0000 7f57367fc700 1 -- 172.21.15.178:0/3693914849 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5730014420 con 0x7f57400f4b00 2023-11-15T04:56:34.063 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:34.062+0000 7f5748212700 1 -- 172.21.15.178:0/3693914849 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57400fa5c0 con 0x7f57400f4b00 2023-11-15T04:56:34.064 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:34.063+0000 7f57367fc700 1 -- 172.21.15.178:0/3693914849 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f573001d7f0 con 0x7f57400f4b00 2023-11-15T04:56:34.064 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:56:34.065 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:56:34.066 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:34.064+0000 7f5748212700 1 -- 172.21.15.178:0/3693914849 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f572c02ff60 msgr2=0x7f572c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:34.066 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:34.064+0000 7f5748212700 1 --2- 172.21.15.178:0/3693914849 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f572c02ff60 0x7f572c032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f573c0096a0 tx=0x7f573c006b40).stop 2023-11-15T04:56:34.066 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:34.064+0000 7f5748212700 1 -- 172.21.15.178:0/3693914849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57400f4b00 msgr2=0x7f5740106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:34.067 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:34.064+0000 7f5748212700 1 --2- 172.21.15.178:0/3693914849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57400f4b00 0x7f5740106710 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f5730013510 tx=0x7f573000b040).stop 2023-11-15T04:56:34.067 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:34.064+0000 7f5748212700 1 -- 172.21.15.178:0/3693914849 shutdown_connections 2023-11-15T04:56:34.067 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:34.064+0000 7f5748212700 1 --2- 172.21.15.178:0/3693914849 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f572c02ff60 0x7f572c032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:34.067 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:34.064+0000 7f5748212700 1 --2- 172.21.15.178:0/3693914849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57400f4b00 0x7f5740106710 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:34.067 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:34.064+0000 7f5748212700 1 -- 172.21.15.178:0/3693914849 >> 172.21.15.178:0/3693914849 conn(0x7f57400f0260 msgr2=0x7f57400f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:34.068 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:34.064+0000 7f5748212700 1 -- 172.21.15.178:0/3693914849 shutdown_connections 2023-11-15T04:56:34.068 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:34.064+0000 7f5748212700 1 -- 172.21.15.178:0/3693914849 wait complete. 2023-11-15T04:56:34.069 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:56:34.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:33 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:32.317850+0000 mgr.smithi163.qesyrx (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:35.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:34 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:34.064646+0000 mon.smithi163 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.178:0/3693914849' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:56:35.444 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:56:35.445 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:56:36.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:34.318202+0000 mgr.smithi163.qesyrx (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:36.575 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.574+0000 7f5a2ffff700 1 -- 172.21.15.178:0/3793881354 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a300f62a0 msgr2=0x7f5a300f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:36.576 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.574+0000 7f5a2ffff700 1 --2- 172.21.15.178:0/3793881354 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a300f62a0 0x7f5a300f66c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f5a20009fb0 tx=0x7f5a20009b30).stop 2023-11-15T04:56:36.576 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.574+0000 7f5a2ffff700 1 -- 172.21.15.178:0/3793881354 shutdown_connections 2023-11-15T04:56:36.576 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.574+0000 7f5a2ffff700 1 --2- 172.21.15.178:0/3793881354 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a300f62a0 0x7f5a300f66c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:36.576 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.574+0000 7f5a2ffff700 1 -- 172.21.15.178:0/3793881354 >> 172.21.15.178:0/3793881354 conn(0x7f5a300f1ac0 msgr2=0x7f5a300f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:36.577 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.574+0000 7f5a2ffff700 1 -- 172.21.15.178:0/3793881354 shutdown_connections 2023-11-15T04:56:36.577 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.574+0000 7f5a2ffff700 1 -- 172.21.15.178:0/3793881354 wait complete. 2023-11-15T04:56:36.579 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.575+0000 7f5a2ffff700 1 Processor -- start 2023-11-15T04:56:36.579 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.575+0000 7f5a2ffff700 1 -- start start 2023-11-15T04:56:36.579 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.575+0000 7f5a2ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a300f62a0 0x7f5a30106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:36.579 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.575+0000 7f5a2ffff700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a301069f0 con 0x7f5a300f62a0 2023-11-15T04:56:36.579 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.575+0000 7f5a2effd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a300f62a0 0x7f5a30106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:36.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.575+0000 7f5a2effd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a300f62a0 0x7f5a30106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54998/0 (socket says 172.21.15.178:54998) 2023-11-15T04:56:36.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.575+0000 7f5a2effd700 1 -- 172.21.15.178:0/1175189751 learned_addr learned my addr 172.21.15.178:0/1175189751 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:56:36.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.576+0000 7f5a2effd700 1 -- 172.21.15.178:0/1175189751 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a20005040 con 0x7f5a300f62a0 2023-11-15T04:56:36.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.576+0000 7f5a2effd700 1 --2- 172.21.15.178:0/1175189751 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a300f62a0 0x7f5a30106500 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f5a20009830 tx=0x7f5a2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:36.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.576+0000 7f5a177fe700 1 -- 172.21.15.178:0/1175189751 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a20009320 con 0x7f5a300f62a0 2023-11-15T04:56:36.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.576+0000 7f5a2ffff700 1 -- 172.21.15.178:0/1175189751 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a30106bf0 con 0x7f5a300f62a0 2023-11-15T04:56:36.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.576+0000 7f5a177fe700 1 -- 172.21.15.178:0/1175189751 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a200136c0 con 0x7f5a300f62a0 2023-11-15T04:56:36.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.576+0000 7f5a177fe700 1 -- 172.21.15.178:0/1175189751 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a2001ee90 con 0x7f5a300f62a0 2023-11-15T04:56:36.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.576+0000 7f5a2ffff700 1 -- 172.21.15.178:0/1175189751 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a30107080 con 0x7f5a300f62a0 2023-11-15T04:56:36.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.577+0000 7f5a2ffff700 1 -- 172.21.15.178:0/1175189751 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a30045ad0 con 0x7f5a300f62a0 2023-11-15T04:56:36.582 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.577+0000 7f5a177fe700 1 -- 172.21.15.178:0/1175189751 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5a2001e420 con 0x7f5a300f62a0 2023-11-15T04:56:36.582 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.577+0000 7f5a177fe700 1 --2- 172.21.15.178:0/1175189751 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5a1802ff10 0x7f5a180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:36.582 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.577+0000 7f5a177fe700 1 -- 172.21.15.178:0/1175189751 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5a20041050 con 0x7f5a300f62a0 2023-11-15T04:56:36.582 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.577+0000 7f5a2e7fc700 1 --2- 172.21.15.178:0/1175189751 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5a1802ff10 0x7f5a180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:36.582 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.578+0000 7f5a2e7fc700 1 --2- 172.21.15.178:0/1175189751 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5a1802ff10 0x7f5a180323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f5a240096a0 tx=0x7f5a24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:36.583 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.582+0000 7f5a177fe700 1 -- 172.21.15.178:0/1175189751 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a2006d2b0 con 0x7f5a300f62a0 2023-11-15T04:56:36.791 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.790+0000 7f5a2ffff700 1 -- 172.21.15.178:0/1175189751 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a30107ed0 con 0x7f5a300f62a0 2023-11-15T04:56:36.792 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.791+0000 7f5a177fe700 1 -- 172.21.15.178:0/1175189751 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5a200095f0 con 0x7f5a300f62a0 2023-11-15T04:56:36.792 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:56:36.792 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:56:36.794 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.792+0000 7f5a2ffff700 1 -- 172.21.15.178:0/1175189751 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5a1802ff10 msgr2=0x7f5a180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:36.794 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.792+0000 7f5a2ffff700 1 --2- 172.21.15.178:0/1175189751 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5a1802ff10 0x7f5a180323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f5a240096a0 tx=0x7f5a24006b40).stop 2023-11-15T04:56:36.794 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.792+0000 7f5a2ffff700 1 -- 172.21.15.178:0/1175189751 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a300f62a0 msgr2=0x7f5a30106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:36.794 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.792+0000 7f5a2ffff700 1 --2- 172.21.15.178:0/1175189751 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a300f62a0 0x7f5a30106500 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f5a20009830 tx=0x7f5a2000b040).stop 2023-11-15T04:56:36.795 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.792+0000 7f5a2ffff700 1 -- 172.21.15.178:0/1175189751 shutdown_connections 2023-11-15T04:56:36.795 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.792+0000 7f5a2ffff700 1 --2- 172.21.15.178:0/1175189751 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5a1802ff10 0x7f5a180323d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:36.795 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.792+0000 7f5a2ffff700 1 --2- 172.21.15.178:0/1175189751 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a300f62a0 0x7f5a30106500 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:36.795 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.792+0000 7f5a2ffff700 1 -- 172.21.15.178:0/1175189751 >> 172.21.15.178:0/1175189751 conn(0x7f5a300f1ac0 msgr2=0x7f5a300f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:36.795 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.792+0000 7f5a2ffff700 1 -- 172.21.15.178:0/1175189751 shutdown_connections 2023-11-15T04:56:36.796 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:36.792+0000 7f5a2ffff700 1 -- 172.21.15.178:0/1175189751 wait complete. 2023-11-15T04:56:36.796 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:56:38.163 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:56:38.164 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:56:38.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:37 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:36.318568+0000 mgr.smithi163.qesyrx (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:38.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:37 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:36.792216+0000 mon.smithi163 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.178:0/1175189751' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:56:39.329 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.328+0000 7f0e0a80c700 1 -- 172.21.15.178:0/1505718666 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e040f6510 msgr2=0x7f0e040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:39.330 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.328+0000 7f0e0a80c700 1 --2- 172.21.15.178:0/1505718666 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e040f6510 0x7f0e040f6930 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f0df8009fb0 tx=0x7f0df8009b30).stop 2023-11-15T04:56:39.330 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.328+0000 7f0e0a80c700 1 -- 172.21.15.178:0/1505718666 shutdown_connections 2023-11-15T04:56:39.330 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.328+0000 7f0e0a80c700 1 --2- 172.21.15.178:0/1505718666 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e040f6510 0x7f0e040f6930 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:39.331 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.328+0000 7f0e0a80c700 1 -- 172.21.15.178:0/1505718666 >> 172.21.15.178:0/1505718666 conn(0x7f0e040f1c70 msgr2=0x7f0e040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:39.331 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.328+0000 7f0e0a80c700 1 -- 172.21.15.178:0/1505718666 shutdown_connections 2023-11-15T04:56:39.331 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.328+0000 7f0e0a80c700 1 -- 172.21.15.178:0/1505718666 wait complete. 2023-11-15T04:56:39.333 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.329+0000 7f0e0a80c700 1 Processor -- start 2023-11-15T04:56:39.333 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.329+0000 7f0e0a80c700 1 -- start start 2023-11-15T04:56:39.334 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.329+0000 7f0e0a80c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e040f6510 0x7f0e04106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:39.334 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.329+0000 7f0e0a80c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e04106c20 con 0x7f0e040f6510 2023-11-15T04:56:39.334 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.329+0000 7f0e03fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e040f6510 0x7f0e04106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:39.334 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.329+0000 7f0e03fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e040f6510 0x7f0e04106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:55024/0 (socket says 172.21.15.178:55024) 2023-11-15T04:56:39.334 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.329+0000 7f0e03fff700 1 -- 172.21.15.178:0/2581173128 learned_addr learned my addr 172.21.15.178:0/2581173128 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:56:39.335 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.330+0000 7f0e03fff700 1 -- 172.21.15.178:0/2581173128 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df8005040 con 0x7f0e040f6510 2023-11-15T04:56:39.335 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.330+0000 7f0e03fff700 1 --2- 172.21.15.178:0/2581173128 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e040f6510 0x7f0e04106730 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f0df8013510 tx=0x7f0df800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:39.335 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.330+0000 7f0e00ff9700 1 -- 172.21.15.178:0/2581173128 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0df8009320 con 0x7f0e040f6510 2023-11-15T04:56:39.335 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.330+0000 7f0e00ff9700 1 -- 172.21.15.178:0/2581173128 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0df8013f00 con 0x7f0e040f6510 2023-11-15T04:56:39.335 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.330+0000 7f0e0a80c700 1 -- 172.21.15.178:0/2581173128 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e04106e20 con 0x7f0e040f6510 2023-11-15T04:56:39.336 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.330+0000 7f0e00ff9700 1 -- 172.21.15.178:0/2581173128 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0df8014d90 con 0x7f0e040f6510 2023-11-15T04:56:39.336 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.330+0000 7f0e0a80c700 1 -- 172.21.15.178:0/2581173128 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e041072b0 con 0x7f0e040f6510 2023-11-15T04:56:39.336 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.331+0000 7f0e00ff9700 1 -- 172.21.15.178:0/2581173128 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0df80263a0 con 0x7f0e040f6510 2023-11-15T04:56:39.336 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.331+0000 7f0e00ff9700 1 --2- 172.21.15.178:0/2581173128 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f0dec02ff60 0x7f0dec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:39.336 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.331+0000 7f0e00ff9700 1 -- 172.21.15.178:0/2581173128 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0df8041020 con 0x7f0e040f6510 2023-11-15T04:56:39.336 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.331+0000 7f0e0a80c700 1 -- 172.21.15.178:0/2581173128 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e04059100 con 0x7f0e040f6510 2023-11-15T04:56:39.337 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.331+0000 7f0e037fe700 1 --2- 172.21.15.178:0/2581173128 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f0dec02ff60 0x7f0dec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:39.337 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.332+0000 7f0e037fe700 1 --2- 172.21.15.178:0/2581173128 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f0dec02ff60 0x7f0dec032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f0df40096a0 tx=0x7f0df4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:39.337 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.336+0000 7f0e00ff9700 1 -- 172.21.15.178:0/2581173128 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0df8012920 con 0x7f0e040f6510 2023-11-15T04:56:39.544 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.543+0000 7f0e0a80c700 1 -- 172.21.15.178:0/2581173128 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e040fbf60 con 0x7f0e040f6510 2023-11-15T04:56:39.545 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.544+0000 7f0e00ff9700 1 -- 172.21.15.178:0/2581173128 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0df802b020 con 0x7f0e040f6510 2023-11-15T04:56:39.546 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:56:39.546 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:56:39.547 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.545+0000 7f0e0a80c700 1 -- 172.21.15.178:0/2581173128 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f0dec02ff60 msgr2=0x7f0dec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:39.547 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.545+0000 7f0e0a80c700 1 --2- 172.21.15.178:0/2581173128 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f0dec02ff60 0x7f0dec032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f0df40096a0 tx=0x7f0df4006b40).stop 2023-11-15T04:56:39.547 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.545+0000 7f0e0a80c700 1 -- 172.21.15.178:0/2581173128 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e040f6510 msgr2=0x7f0e04106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:39.547 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.545+0000 7f0e0a80c700 1 --2- 172.21.15.178:0/2581173128 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e040f6510 0x7f0e04106730 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f0df8013510 tx=0x7f0df800b040).stop 2023-11-15T04:56:39.547 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.545+0000 7f0e0a80c700 1 -- 172.21.15.178:0/2581173128 shutdown_connections 2023-11-15T04:56:39.548 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.545+0000 7f0e0a80c700 1 --2- 172.21.15.178:0/2581173128 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f0dec02ff60 0x7f0dec032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:39.548 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.545+0000 7f0e0a80c700 1 --2- 172.21.15.178:0/2581173128 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e040f6510 0x7f0e04106730 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:39.548 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.545+0000 7f0e0a80c700 1 -- 172.21.15.178:0/2581173128 >> 172.21.15.178:0/2581173128 conn(0x7f0e040f1c70 msgr2=0x7f0e040f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:39.548 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.545+0000 7f0e0a80c700 1 -- 172.21.15.178:0/2581173128 shutdown_connections 2023-11-15T04:56:39.548 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:39.545+0000 7f0e0a80c700 1 -- 172.21.15.178:0/2581173128 wait complete. 2023-11-15T04:56:39.549 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:56:40.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:39 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:38.318926+0000 mgr.smithi163.qesyrx (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:40.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:39 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:39.545576+0000 mon.smithi163 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.178:0/2581173128' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:56:40.921 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:56:40.922 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:56:42.020 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.018+0000 7fbac9d1f700 1 -- 172.21.15.178:0/1774577257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbac40f51a0 msgr2=0x7fbac40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:42.021 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.018+0000 7fbac9d1f700 1 --2- 172.21.15.178:0/1774577257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbac40f51a0 0x7fbac40f55c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fbab8009fb0 tx=0x7fbab8009b30).stop 2023-11-15T04:56:42.021 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.018+0000 7fbac9d1f700 1 -- 172.21.15.178:0/1774577257 shutdown_connections 2023-11-15T04:56:42.021 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.018+0000 7fbac9d1f700 1 --2- 172.21.15.178:0/1774577257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbac40f51a0 0x7fbac40f55c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:42.021 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.018+0000 7fbac9d1f700 1 -- 172.21.15.178:0/1774577257 >> 172.21.15.178:0/1774577257 conn(0x7fbac40f09a0 msgr2=0x7fbac40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:42.021 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.018+0000 7fbac9d1f700 1 -- 172.21.15.178:0/1774577257 shutdown_connections 2023-11-15T04:56:42.022 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.018+0000 7fbac9d1f700 1 -- 172.21.15.178:0/1774577257 wait complete. 2023-11-15T04:56:42.022 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.019+0000 7fbac9d1f700 1 Processor -- start 2023-11-15T04:56:42.022 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.019+0000 7fbac9d1f700 1 -- start start 2023-11-15T04:56:42.024 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.020+0000 7fbac9d1f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbac40f51a0 0x7fbac41053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:42.024 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.020+0000 7fbac9d1f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac41058a0 con 0x7fbac40f51a0 2023-11-15T04:56:42.024 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.020+0000 7fbac8d1d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbac40f51a0 0x7fbac41053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:42.025 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.020+0000 7fbac8d1d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbac40f51a0 0x7fbac41053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:55042/0 (socket says 172.21.15.178:55042) 2023-11-15T04:56:42.025 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.020+0000 7fbac8d1d700 1 -- 172.21.15.178:0/729920871 learned_addr learned my addr 172.21.15.178:0/729920871 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:56:42.025 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.020+0000 7fbac8d1d700 1 -- 172.21.15.178:0/729920871 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab8005040 con 0x7fbac40f51a0 2023-11-15T04:56:42.025 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.020+0000 7fbac8d1d700 1 --2- 172.21.15.178:0/729920871 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbac40f51a0 0x7fbac41053b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fbab8013510 tx=0x7fbab800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:42.026 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.020+0000 7fbac17fa700 1 -- 172.21.15.178:0/729920871 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbab8009320 con 0x7fbac40f51a0 2023-11-15T04:56:42.026 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.020+0000 7fbac9d1f700 1 -- 172.21.15.178:0/729920871 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac4105aa0 con 0x7fbac40f51a0 2023-11-15T04:56:42.026 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.020+0000 7fbac17fa700 1 -- 172.21.15.178:0/729920871 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbab8013f00 con 0x7fbac40f51a0 2023-11-15T04:56:42.026 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.020+0000 7fbac17fa700 1 -- 172.21.15.178:0/729920871 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbab8014dc0 con 0x7fbac40f51a0 2023-11-15T04:56:42.027 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.020+0000 7fbac9d1f700 1 -- 172.21.15.178:0/729920871 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac4105f30 con 0x7fbac40f51a0 2023-11-15T04:56:42.027 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.021+0000 7fbac17fa700 1 -- 172.21.15.178:0/729920871 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbab80095f0 con 0x7fbac40f51a0 2023-11-15T04:56:42.027 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.021+0000 7fbac9d1f700 1 -- 172.21.15.178:0/729920871 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac4045ad0 con 0x7fbac40f51a0 2023-11-15T04:56:42.027 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.021+0000 7fbac17fa700 1 --2- 172.21.15.178:0/729920871 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fbab402ff60 0x7fbab4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:42.027 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.021+0000 7fbac17fa700 1 -- 172.21.15.178:0/729920871 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbab8041040 con 0x7fbac40f51a0 2023-11-15T04:56:42.028 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.022+0000 7fbac3fff700 1 --2- 172.21.15.178:0/729920871 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fbab402ff60 0x7fbab4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:42.028 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.022+0000 7fbac3fff700 1 --2- 172.21.15.178:0/729920871 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fbab402ff60 0x7fbab4032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fbab00096a0 tx=0x7fbab0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:42.028 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.026+0000 7fbac17fa700 1 -- 172.21.15.178:0/729920871 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbab8014420 con 0x7fbac40f51a0 2023-11-15T04:56:42.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:41 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:40.319256+0000 mgr.smithi163.qesyrx (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:42.234 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.233+0000 7fbac9d1f700 1 -- 172.21.15.178:0/729920871 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbac4106de0 con 0x7fbac40f51a0 2023-11-15T04:56:42.235 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.234+0000 7fbac17fa700 1 -- 172.21.15.178:0/729920871 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbab8010230 con 0x7fbac40f51a0 2023-11-15T04:56:42.235 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:56:42.236 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:56:42.237 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.235+0000 7fbac9d1f700 1 -- 172.21.15.178:0/729920871 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fbab402ff60 msgr2=0x7fbab4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:42.237 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.235+0000 7fbac9d1f700 1 --2- 172.21.15.178:0/729920871 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fbab402ff60 0x7fbab4032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fbab00096a0 tx=0x7fbab0006b40).stop 2023-11-15T04:56:42.237 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.235+0000 7fbac9d1f700 1 -- 172.21.15.178:0/729920871 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbac40f51a0 msgr2=0x7fbac41053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:42.237 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.235+0000 7fbac9d1f700 1 --2- 172.21.15.178:0/729920871 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbac40f51a0 0x7fbac41053b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fbab8013510 tx=0x7fbab800b040).stop 2023-11-15T04:56:42.237 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.235+0000 7fbac9d1f700 1 -- 172.21.15.178:0/729920871 shutdown_connections 2023-11-15T04:56:42.238 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.235+0000 7fbac9d1f700 1 --2- 172.21.15.178:0/729920871 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fbab402ff60 0x7fbab4032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:42.238 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.235+0000 7fbac9d1f700 1 --2- 172.21.15.178:0/729920871 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbac40f51a0 0x7fbac41053b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:42.238 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.235+0000 7fbac9d1f700 1 -- 172.21.15.178:0/729920871 >> 172.21.15.178:0/729920871 conn(0x7fbac40f09a0 msgr2=0x7fbac40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:42.239 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.235+0000 7fbac9d1f700 1 -- 172.21.15.178:0/729920871 shutdown_connections 2023-11-15T04:56:42.239 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:42.235+0000 7fbac9d1f700 1 -- 172.21.15.178:0/729920871 wait complete. 2023-11-15T04:56:42.241 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:56:43.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:42 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:42.235269+0000 mon.smithi163 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.178:0/729920871' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:56:43.611 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:56:43.612 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:56:44.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:43 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:42.319630+0000 mgr.smithi163.qesyrx (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:44.799 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.797+0000 7f69b2c05700 1 -- 172.21.15.178:0/1681832035 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69ac0f6510 msgr2=0x7f69ac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:44.799 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.797+0000 7f69b2c05700 1 --2- 172.21.15.178:0/1681832035 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69ac0f6510 0x7f69ac0f6930 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f69a0009fb0 tx=0x7f69a0009b30).stop 2023-11-15T04:56:44.799 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.797+0000 7f69b2c05700 1 -- 172.21.15.178:0/1681832035 shutdown_connections 2023-11-15T04:56:44.800 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.797+0000 7f69b2c05700 1 --2- 172.21.15.178:0/1681832035 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69ac0f6510 0x7f69ac0f6930 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:44.800 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.797+0000 7f69b2c05700 1 -- 172.21.15.178:0/1681832035 >> 172.21.15.178:0/1681832035 conn(0x7f69ac0f1c70 msgr2=0x7f69ac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:44.800 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.797+0000 7f69b2c05700 1 -- 172.21.15.178:0/1681832035 shutdown_connections 2023-11-15T04:56:44.800 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.797+0000 7f69b2c05700 1 -- 172.21.15.178:0/1681832035 wait complete. 2023-11-15T04:56:44.800 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.797+0000 7f69b2c05700 1 Processor -- start 2023-11-15T04:56:44.801 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.797+0000 7f69b2c05700 1 -- start start 2023-11-15T04:56:44.803 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.798+0000 7f69b2c05700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69ac0f6510 0x7f69ac106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:44.803 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.798+0000 7f69b2c05700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac106c20 con 0x7f69ac0f6510 2023-11-15T04:56:44.803 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.798+0000 7f69b09a1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69ac0f6510 0x7f69ac106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:44.803 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.798+0000 7f69b09a1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69ac0f6510 0x7f69ac106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:34360/0 (socket says 172.21.15.178:34360) 2023-11-15T04:56:44.804 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.798+0000 7f69b09a1700 1 -- 172.21.15.178:0/323135022 learned_addr learned my addr 172.21.15.178:0/323135022 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:56:44.804 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.799+0000 7f69b09a1700 1 -- 172.21.15.178:0/323135022 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a0005040 con 0x7f69ac0f6510 2023-11-15T04:56:44.804 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.799+0000 7f69b09a1700 1 --2- 172.21.15.178:0/323135022 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69ac0f6510 0x7f69ac106730 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f69a0013510 tx=0x7f69a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:44.804 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.799+0000 7f69a97fa700 1 -- 172.21.15.178:0/323135022 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69a0009320 con 0x7f69ac0f6510 2023-11-15T04:56:44.805 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.799+0000 7f69a97fa700 1 -- 172.21.15.178:0/323135022 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69a0013f00 con 0x7f69ac0f6510 2023-11-15T04:56:44.805 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.799+0000 7f69b2c05700 1 -- 172.21.15.178:0/323135022 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69ac106e20 con 0x7f69ac0f6510 2023-11-15T04:56:44.805 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.799+0000 7f69a97fa700 1 -- 172.21.15.178:0/323135022 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69a0014d90 con 0x7f69ac0f6510 2023-11-15T04:56:44.805 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.799+0000 7f69b2c05700 1 -- 172.21.15.178:0/323135022 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69ac1072b0 con 0x7f69ac0f6510 2023-11-15T04:56:44.806 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.800+0000 7f69a97fa700 1 -- 172.21.15.178:0/323135022 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69a00263a0 con 0x7f69ac0f6510 2023-11-15T04:56:44.806 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.800+0000 7f69b2c05700 1 -- 172.21.15.178:0/323135022 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69ac045ad0 con 0x7f69ac0f6510 2023-11-15T04:56:44.806 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.800+0000 7f69a97fa700 1 --2- 172.21.15.178:0/323135022 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f699c02ff60 0x7f699c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:44.806 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.800+0000 7f69a97fa700 1 -- 172.21.15.178:0/323135022 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f69a00412f0 con 0x7f69ac0f6510 2023-11-15T04:56:44.807 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.800+0000 7f69abfff700 1 --2- 172.21.15.178:0/323135022 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f699c02ff60 0x7f699c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:44.807 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.800+0000 7f69abfff700 1 --2- 172.21.15.178:0/323135022 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f699c02ff60 0x7f699c032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f69980096a0 tx=0x7f6998006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:44.807 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:44.805+0000 7f69a97fa700 1 -- 172.21.15.178:0/323135022 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69a001cad0 con 0x7f69ac0f6510 2023-11-15T04:56:45.011 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:45.010+0000 7f69b2c05700 1 -- 172.21.15.178:0/323135022 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69ac0fbf60 con 0x7f69ac0f6510 2023-11-15T04:56:45.012 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:45.010+0000 7f69a97fa700 1 -- 172.21.15.178:0/323135022 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f69a001cad0 con 0x7f69ac0f6510 2023-11-15T04:56:45.012 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:56:45.012 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:56:45.014 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:45.012+0000 7f69b2c05700 1 -- 172.21.15.178:0/323135022 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f699c02ff60 msgr2=0x7f699c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:45.014 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:45.012+0000 7f69b2c05700 1 --2- 172.21.15.178:0/323135022 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f699c02ff60 0x7f699c032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f69980096a0 tx=0x7f6998006b40).stop 2023-11-15T04:56:45.014 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:45.012+0000 7f69b2c05700 1 -- 172.21.15.178:0/323135022 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69ac0f6510 msgr2=0x7f69ac106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:45.014 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:45.012+0000 7f69b2c05700 1 --2- 172.21.15.178:0/323135022 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69ac0f6510 0x7f69ac106730 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f69a0013510 tx=0x7f69a000b040).stop 2023-11-15T04:56:45.014 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:45.012+0000 7f69b2c05700 1 -- 172.21.15.178:0/323135022 shutdown_connections 2023-11-15T04:56:45.015 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:45.012+0000 7f69b2c05700 1 --2- 172.21.15.178:0/323135022 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f699c02ff60 0x7f699c032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:45.015 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:45.012+0000 7f69b2c05700 1 --2- 172.21.15.178:0/323135022 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69ac0f6510 0x7f69ac106730 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:45.015 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:45.012+0000 7f69b2c05700 1 -- 172.21.15.178:0/323135022 >> 172.21.15.178:0/323135022 conn(0x7f69ac0f1c70 msgr2=0x7f69ac0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:45.015 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:45.012+0000 7f69b2c05700 1 -- 172.21.15.178:0/323135022 shutdown_connections 2023-11-15T04:56:45.016 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:45.012+0000 7f69b2c05700 1 -- 172.21.15.178:0/323135022 wait complete. 2023-11-15T04:56:45.016 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:56:46.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:45 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:44.320001+0000 mgr.smithi163.qesyrx (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:46.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:45 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:45.011937+0000 mon.smithi163 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.178:0/323135022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:56:46.390 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:56:46.391 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:56:47.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.486+0000 7fef06ad8700 1 -- 172.21.15.178:0/296506860 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef000f5410 msgr2=0x7fef000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:47.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.486+0000 7fef06ad8700 1 --2- 172.21.15.178:0/296506860 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef000f5410 0x7fef000f5830 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7feef4009fb0 tx=0x7feef4009b30).stop 2023-11-15T04:56:47.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.486+0000 7fef06ad8700 1 -- 172.21.15.178:0/296506860 shutdown_connections 2023-11-15T04:56:47.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.486+0000 7fef06ad8700 1 --2- 172.21.15.178:0/296506860 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef000f5410 0x7fef000f5830 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:47.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.486+0000 7fef06ad8700 1 -- 172.21.15.178:0/296506860 >> 172.21.15.178:0/296506860 conn(0x7fef000f0b70 msgr2=0x7fef000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:47.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.486+0000 7fef06ad8700 1 -- 172.21.15.178:0/296506860 shutdown_connections 2023-11-15T04:56:47.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.486+0000 7fef06ad8700 1 -- 172.21.15.178:0/296506860 wait complete. 2023-11-15T04:56:47.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.487+0000 7fef06ad8700 1 Processor -- start 2023-11-15T04:56:47.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.487+0000 7fef06ad8700 1 -- start start 2023-11-15T04:56:47.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.487+0000 7fef06ad8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef000f5410 0x7fef00109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:47.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.487+0000 7fef06ad8700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef00060220 con 0x7fef000f5410 2023-11-15T04:56:47.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.487+0000 7fef04874700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef000f5410 0x7fef00109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:47.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.487+0000 7fef04874700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef000f5410 0x7fef00109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:34372/0 (socket says 172.21.15.178:34372) 2023-11-15T04:56:47.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.487+0000 7fef04874700 1 -- 172.21.15.178:0/3710276837 learned_addr learned my addr 172.21.15.178:0/3710276837 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:56:47.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.488+0000 7fef04874700 1 -- 172.21.15.178:0/3710276837 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef4005040 con 0x7fef000f5410 2023-11-15T04:56:47.492 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.488+0000 7fef04874700 1 --2- 172.21.15.178:0/3710276837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef000f5410 0x7fef00109420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7feef4013510 tx=0x7feef400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:47.492 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.488+0000 7feefd7fa700 1 -- 172.21.15.178:0/3710276837 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feef4009320 con 0x7fef000f5410 2023-11-15T04:56:47.492 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.488+0000 7feefd7fa700 1 -- 172.21.15.178:0/3710276837 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feef4013f00 con 0x7fef000f5410 2023-11-15T04:56:47.492 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.488+0000 7fef06ad8700 1 -- 172.21.15.178:0/3710276837 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef00060420 con 0x7fef000f5410 2023-11-15T04:56:47.492 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.488+0000 7feefd7fa700 1 -- 172.21.15.178:0/3710276837 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feef4014d90 con 0x7fef000f5410 2023-11-15T04:56:47.493 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.488+0000 7fef06ad8700 1 -- 172.21.15.178:0/3710276837 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef00109ad0 con 0x7fef000f5410 2023-11-15T04:56:47.494 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.489+0000 7feefd7fa700 1 -- 172.21.15.178:0/3710276837 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feef4014420 con 0x7fef000f5410 2023-11-15T04:56:47.494 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.489+0000 7fef06ad8700 1 -- 172.21.15.178:0/3710276837 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef00059100 con 0x7fef000f5410 2023-11-15T04:56:47.494 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.489+0000 7feefd7fa700 1 --2- 172.21.15.178:0/3710276837 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feef002ff60 0x7feef0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:47.494 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.489+0000 7feefd7fa700 1 -- 172.21.15.178:0/3710276837 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feef40410c0 con 0x7fef000f5410 2023-11-15T04:56:47.495 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.489+0000 7feefffff700 1 --2- 172.21.15.178:0/3710276837 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feef002ff60 0x7feef0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:47.495 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.489+0000 7feefffff700 1 --2- 172.21.15.178:0/3710276837 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feef002ff60 0x7feef0032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7feeec0096a0 tx=0x7feeec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:47.495 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.494+0000 7feefd7fa700 1 -- 172.21.15.178:0/3710276837 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feef40095f0 con 0x7fef000f5410 2023-11-15T04:56:47.705 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.704+0000 7fef06ad8700 1 -- 172.21.15.178:0/3710276837 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fef00045ad0 con 0x7fef000f5410 2023-11-15T04:56:47.705 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.705+0000 7feefd7fa700 1 -- 172.21.15.178:0/3710276837 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7feef401db70 con 0x7fef000f5410 2023-11-15T04:56:47.706 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:56:47.706 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:56:47.707 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.706+0000 7fef06ad8700 1 -- 172.21.15.178:0/3710276837 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feef002ff60 msgr2=0x7feef0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:47.707 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.706+0000 7fef06ad8700 1 --2- 172.21.15.178:0/3710276837 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feef002ff60 0x7feef0032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7feeec0096a0 tx=0x7feeec006b40).stop 2023-11-15T04:56:47.707 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.706+0000 7fef06ad8700 1 -- 172.21.15.178:0/3710276837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef000f5410 msgr2=0x7fef00109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:47.709 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.706+0000 7fef06ad8700 1 --2- 172.21.15.178:0/3710276837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef000f5410 0x7fef00109420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7feef4013510 tx=0x7feef400b040).stop 2023-11-15T04:56:47.709 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.706+0000 7fef06ad8700 1 -- 172.21.15.178:0/3710276837 shutdown_connections 2023-11-15T04:56:47.709 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.706+0000 7fef06ad8700 1 --2- 172.21.15.178:0/3710276837 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feef002ff60 0x7feef0032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:47.710 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.706+0000 7fef06ad8700 1 --2- 172.21.15.178:0/3710276837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef000f5410 0x7fef00109420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:47.710 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.706+0000 7fef06ad8700 1 -- 172.21.15.178:0/3710276837 >> 172.21.15.178:0/3710276837 conn(0x7fef000f0b70 msgr2=0x7fef000f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:47.710 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.707+0000 7fef06ad8700 1 -- 172.21.15.178:0/3710276837 shutdown_connections 2023-11-15T04:56:47.710 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:47.707+0000 7fef06ad8700 1 -- 172.21.15.178:0/3710276837 wait complete. 2023-11-15T04:56:47.711 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:56:48.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:47 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:46.320385+0000 mgr.smithi163.qesyrx (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:48.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:47 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:47.705589+0000 mon.smithi163 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.178:0/3710276837' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:56:49.119 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:56:49.120 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:56:50.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:49 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:48.320702+0000 mgr.smithi163.qesyrx (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:50.279 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.279+0000 7feebd1bc700 1 -- 172.21.15.178:0/1098063864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feeb80f6510 msgr2=0x7feeb80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:50.280 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.279+0000 7feebd1bc700 1 --2- 172.21.15.178:0/1098063864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feeb80f6510 0x7feeb80f6930 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7feea0009fb0 tx=0x7feea0009b30).stop 2023-11-15T04:56:50.280 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.279+0000 7feebd1bc700 1 -- 172.21.15.178:0/1098063864 shutdown_connections 2023-11-15T04:56:50.280 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.279+0000 7feebd1bc700 1 --2- 172.21.15.178:0/1098063864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feeb80f6510 0x7feeb80f6930 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:50.280 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.279+0000 7feebd1bc700 1 -- 172.21.15.178:0/1098063864 >> 172.21.15.178:0/1098063864 conn(0x7feeb80f1c70 msgr2=0x7feeb80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:50.281 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.279+0000 7feebd1bc700 1 -- 172.21.15.178:0/1098063864 shutdown_connections 2023-11-15T04:56:50.281 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.279+0000 7feebd1bc700 1 -- 172.21.15.178:0/1098063864 wait complete. 2023-11-15T04:56:50.283 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.280+0000 7feebd1bc700 1 Processor -- start 2023-11-15T04:56:50.283 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.280+0000 7feebd1bc700 1 -- start start 2023-11-15T04:56:50.283 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.280+0000 7feebd1bc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feeb80f6510 0x7feeb806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:50.283 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.280+0000 7feebd1bc700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb8069e20 con 0x7feeb80f6510 2023-11-15T04:56:50.284 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.280+0000 7feeb6d9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feeb80f6510 0x7feeb806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:50.284 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.280+0000 7feeb6d9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feeb80f6510 0x7feeb806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:34388/0 (socket says 172.21.15.178:34388) 2023-11-15T04:56:50.284 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.280+0000 7feeb6d9d700 1 -- 172.21.15.178:0/276614098 learned_addr learned my addr 172.21.15.178:0/276614098 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:56:50.284 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.280+0000 7feeb6d9d700 1 -- 172.21.15.178:0/276614098 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea0005040 con 0x7feeb80f6510 2023-11-15T04:56:50.284 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.280+0000 7feeb6d9d700 1 --2- 172.21.15.178:0/276614098 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feeb80f6510 0x7feeb806b9a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7feea0013510 tx=0x7feea000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:50.285 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.281+0000 7feeaf7fe700 1 -- 172.21.15.178:0/276614098 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feea0009320 con 0x7feeb80f6510 2023-11-15T04:56:50.285 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.281+0000 7feebd1bc700 1 -- 172.21.15.178:0/276614098 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeb8069fc0 con 0x7feeb80f6510 2023-11-15T04:56:50.285 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.281+0000 7feeaf7fe700 1 -- 172.21.15.178:0/276614098 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feea0013f00 con 0x7feeb80f6510 2023-11-15T04:56:50.285 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.281+0000 7feeaf7fe700 1 -- 172.21.15.178:0/276614098 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feea0014d90 con 0x7feeb80f6510 2023-11-15T04:56:50.285 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.281+0000 7feebd1bc700 1 -- 172.21.15.178:0/276614098 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeb806a450 con 0x7feeb80f6510 2023-11-15T04:56:50.286 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.281+0000 7feeaf7fe700 1 -- 172.21.15.178:0/276614098 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feea0014420 con 0x7feeb80f6510 2023-11-15T04:56:50.286 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.281+0000 7feebd1bc700 1 -- 172.21.15.178:0/276614098 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeb8059100 con 0x7feeb80f6510 2023-11-15T04:56:50.286 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.282+0000 7feeaf7fe700 1 --2- 172.21.15.178:0/276614098 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feea402ff60 0x7feea4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:50.286 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.282+0000 7feeaf7fe700 1 -- 172.21.15.178:0/276614098 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feea0040ff0 con 0x7feeb80f6510 2023-11-15T04:56:50.286 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.282+0000 7feeb659c700 1 --2- 172.21.15.178:0/276614098 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feea402ff60 0x7feea4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:50.286 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.282+0000 7feeb659c700 1 --2- 172.21.15.178:0/276614098 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feea402ff60 0x7feea4032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7feea80096a0 tx=0x7feea8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:50.287 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.286+0000 7feeaf7fe700 1 -- 172.21.15.178:0/276614098 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feea00095f0 con 0x7feeb80f6510 2023-11-15T04:56:50.495 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.495+0000 7feebd1bc700 1 -- 172.21.15.178:0/276614098 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feeb80fbf60 con 0x7feeb80f6510 2023-11-15T04:56:50.496 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.495+0000 7feeaf7fe700 1 -- 172.21.15.178:0/276614098 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7feea001db70 con 0x7feeb80f6510 2023-11-15T04:56:50.496 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:56:50.496 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:56:50.497 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.497+0000 7feebd1bc700 1 -- 172.21.15.178:0/276614098 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feea402ff60 msgr2=0x7feea4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:50.498 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.497+0000 7feebd1bc700 1 --2- 172.21.15.178:0/276614098 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feea402ff60 0x7feea4032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7feea80096a0 tx=0x7feea8006b40).stop 2023-11-15T04:56:50.498 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.497+0000 7feebd1bc700 1 -- 172.21.15.178:0/276614098 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feeb80f6510 msgr2=0x7feeb806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:50.498 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.497+0000 7feebd1bc700 1 --2- 172.21.15.178:0/276614098 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feeb80f6510 0x7feeb806b9a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7feea0013510 tx=0x7feea000b040).stop 2023-11-15T04:56:50.498 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.497+0000 7feebd1bc700 1 -- 172.21.15.178:0/276614098 shutdown_connections 2023-11-15T04:56:50.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.497+0000 7feebd1bc700 1 --2- 172.21.15.178:0/276614098 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feea402ff60 0x7feea4032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:50.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.497+0000 7feebd1bc700 1 --2- 172.21.15.178:0/276614098 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feeb80f6510 0x7feeb806b9a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:50.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.497+0000 7feebd1bc700 1 -- 172.21.15.178:0/276614098 >> 172.21.15.178:0/276614098 conn(0x7feeb80f1c70 msgr2=0x7feeb80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:50.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.497+0000 7feebd1bc700 1 -- 172.21.15.178:0/276614098 shutdown_connections 2023-11-15T04:56:50.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:50.497+0000 7feebd1bc700 1 -- 172.21.15.178:0/276614098 wait complete. 2023-11-15T04:56:50.500 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:56:51.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:50 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:50.496070+0000 mon.smithi163 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.178:0/276614098' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:56:51.881 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:56:51.881 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:56:52.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:51 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:50.321051+0000 mgr.smithi163.qesyrx (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:53.835 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.833+0000 7fcfb67bb700 1 -- 172.21.15.178:0/3112925131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfb00f6430 msgr2=0x7fcfb00f6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:53.836 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.833+0000 7fcfb67bb700 1 --2- 172.21.15.178:0/3112925131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfb00f6430 0x7fcfb00f6850 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fcfa0009fb0 tx=0x7fcfa0009b30).stop 2023-11-15T04:56:53.836 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.833+0000 7fcfb67bb700 1 -- 172.21.15.178:0/3112925131 shutdown_connections 2023-11-15T04:56:53.836 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.833+0000 7fcfb67bb700 1 --2- 172.21.15.178:0/3112925131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfb00f6430 0x7fcfb00f6850 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:53.837 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.833+0000 7fcfb67bb700 1 -- 172.21.15.178:0/3112925131 >> 172.21.15.178:0/3112925131 conn(0x7fcfb00f1b90 msgr2=0x7fcfb00f3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:53.837 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.833+0000 7fcfb67bb700 1 -- 172.21.15.178:0/3112925131 shutdown_connections 2023-11-15T04:56:53.837 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.833+0000 7fcfb67bb700 1 -- 172.21.15.178:0/3112925131 wait complete. 2023-11-15T04:56:53.837 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.834+0000 7fcfb67bb700 1 Processor -- start 2023-11-15T04:56:53.837 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.834+0000 7fcfb67bb700 1 -- start start 2023-11-15T04:56:53.838 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.834+0000 7fcfb67bb700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfb00f6430 0x7fcfb006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:53.838 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.834+0000 7fcfb67bb700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb0069e20 con 0x7fcfb00f6430 2023-11-15T04:56:53.838 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.834+0000 7fcfaffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfb00f6430 0x7fcfb006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:53.838 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.834+0000 7fcfaffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfb00f6430 0x7fcfb006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:34402/0 (socket says 172.21.15.178:34402) 2023-11-15T04:56:53.838 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.834+0000 7fcfaffff700 1 -- 172.21.15.178:0/2768178925 learned_addr learned my addr 172.21.15.178:0/2768178925 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:56:53.839 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.835+0000 7fcfaffff700 1 -- 172.21.15.178:0/2768178925 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa0005040 con 0x7fcfb00f6430 2023-11-15T04:56:53.839 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.835+0000 7fcfaffff700 1 --2- 172.21.15.178:0/2768178925 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfb00f6430 0x7fcfb006b9a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fcfa0014000 tx=0x7fcfa000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:53.839 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.835+0000 7fcfacff9700 1 -- 172.21.15.178:0/2768178925 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfa00095f0 con 0x7fcfb00f6430 2023-11-15T04:56:53.839 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.835+0000 7fcfb67bb700 1 -- 172.21.15.178:0/2768178925 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb0069fc0 con 0x7fcfb00f6430 2023-11-15T04:56:53.839 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.835+0000 7fcfacff9700 1 -- 172.21.15.178:0/2768178925 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfa0007e70 con 0x7fcfb00f6430 2023-11-15T04:56:53.840 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.835+0000 7fcfacff9700 1 -- 172.21.15.178:0/2768178925 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfa0014ea0 con 0x7fcfb00f6430 2023-11-15T04:56:53.840 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.835+0000 7fcfb67bb700 1 -- 172.21.15.178:0/2768178925 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb006a450 con 0x7fcfb00f6430 2023-11-15T04:56:53.841 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.836+0000 7fcfacff9700 1 -- 172.21.15.178:0/2768178925 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcfa00079c0 con 0x7fcfb00f6430 2023-11-15T04:56:53.841 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.836+0000 7fcfb67bb700 1 -- 172.21.15.178:0/2768178925 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb0045ad0 con 0x7fcfb00f6430 2023-11-15T04:56:53.841 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.836+0000 7fcfacff9700 1 --2- 172.21.15.178:0/2768178925 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcf9c02ff60 0x7fcf9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:53.841 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.836+0000 7fcfacff9700 1 -- 172.21.15.178:0/2768178925 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcfa0040920 con 0x7fcfb00f6430 2023-11-15T04:56:53.842 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.836+0000 7fcfaf7fe700 1 --2- 172.21.15.178:0/2768178925 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcf9c02ff60 0x7fcf9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:53.842 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.837+0000 7fcfaf7fe700 1 --2- 172.21.15.178:0/2768178925 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcf9c02ff60 0x7fcf9c032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fcf980096a0 tx=0x7fcf98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:53.842 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:53.841+0000 7fcfacff9700 1 -- 172.21.15.178:0/2768178925 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcfa000f950 con 0x7fcfb00f6430 2023-11-15T04:56:54.045 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:54.045+0000 7fcfb67bb700 1 -- 172.21.15.178:0/2768178925 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcfb005ea50 con 0x7fcfb00f6430 2023-11-15T04:56:54.046 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:54.046+0000 7fcfacff9700 1 -- 172.21.15.178:0/2768178925 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcfa0010230 con 0x7fcfb00f6430 2023-11-15T04:56:54.046 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:56:54.047 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:56:54.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:54.047+0000 7fcfb67bb700 1 -- 172.21.15.178:0/2768178925 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcf9c02ff60 msgr2=0x7fcf9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:54.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:54.047+0000 7fcfb67bb700 1 --2- 172.21.15.178:0/2768178925 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcf9c02ff60 0x7fcf9c032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fcf980096a0 tx=0x7fcf98006b40).stop 2023-11-15T04:56:54.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:54.047+0000 7fcfb67bb700 1 -- 172.21.15.178:0/2768178925 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfb00f6430 msgr2=0x7fcfb006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:54.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:54.047+0000 7fcfb67bb700 1 --2- 172.21.15.178:0/2768178925 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfb00f6430 0x7fcfb006b9a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fcfa0014000 tx=0x7fcfa000b040).stop 2023-11-15T04:56:54.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:54.047+0000 7fcfb67bb700 1 -- 172.21.15.178:0/2768178925 shutdown_connections 2023-11-15T04:56:54.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:54.047+0000 7fcfb67bb700 1 --2- 172.21.15.178:0/2768178925 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcf9c02ff60 0x7fcf9c032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:54.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:54.047+0000 7fcfb67bb700 1 --2- 172.21.15.178:0/2768178925 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfb00f6430 0x7fcfb006b9a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:54.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:54.047+0000 7fcfb67bb700 1 -- 172.21.15.178:0/2768178925 >> 172.21.15.178:0/2768178925 conn(0x7fcfb00f1b90 msgr2=0x7fcfb00f2630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:54.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:54.047+0000 7fcfb67bb700 1 -- 172.21.15.178:0/2768178925 shutdown_connections 2023-11-15T04:56:54.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:54.047+0000 7fcfb67bb700 1 -- 172.21.15.178:0/2768178925 wait complete. 2023-11-15T04:56:54.050 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:56:54.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:53 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:52.321440+0000 mgr.smithi163.qesyrx (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:55.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:54 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:54.046323+0000 mon.smithi163 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.178:0/2768178925' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:56:55.427 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:56:55.427 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:56:56.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:55 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:54.321717+0000 mgr.smithi163.qesyrx (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:56.613 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.613+0000 7fb9ed088700 1 -- 172.21.15.178:0/2503192585 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9e80f4880 msgr2=0x7fb9e80f4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:56.614 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.613+0000 7fb9ed088700 1 --2- 172.21.15.178:0/2503192585 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9e80f4880 0x7fb9e80f4ca0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fb9dc009fb0 tx=0x7fb9dc009b30).stop 2023-11-15T04:56:56.614 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.613+0000 7fb9ed088700 1 -- 172.21.15.178:0/2503192585 shutdown_connections 2023-11-15T04:56:56.614 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.613+0000 7fb9ed088700 1 --2- 172.21.15.178:0/2503192585 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9e80f4880 0x7fb9e80f4ca0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:56.614 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.613+0000 7fb9ed088700 1 -- 172.21.15.178:0/2503192585 >> 172.21.15.178:0/2503192585 conn(0x7fb9e80f02e0 msgr2=0x7fb9e80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:56.615 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.613+0000 7fb9ed088700 1 -- 172.21.15.178:0/2503192585 shutdown_connections 2023-11-15T04:56:56.615 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.613+0000 7fb9ed088700 1 -- 172.21.15.178:0/2503192585 wait complete. 2023-11-15T04:56:56.617 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.614+0000 7fb9ed088700 1 Processor -- start 2023-11-15T04:56:56.617 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.614+0000 7fb9ed088700 1 -- start start 2023-11-15T04:56:56.617 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.614+0000 7fb9ed088700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9e80f4880 0x7fb9e80f7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:56.617 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.614+0000 7fb9ed088700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e80f58f0 con 0x7fb9e80f4880 2023-11-15T04:56:56.618 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.614+0000 7fb9e6d9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9e80f4880 0x7fb9e80f7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:56.618 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.614+0000 7fb9e6d9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9e80f4880 0x7fb9e80f7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:34726/0 (socket says 172.21.15.178:34726) 2023-11-15T04:56:56.618 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.614+0000 7fb9e6d9d700 1 -- 172.21.15.178:0/796852307 learned_addr learned my addr 172.21.15.178:0/796852307 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:56:56.618 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.615+0000 7fb9e6d9d700 1 -- 172.21.15.178:0/796852307 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc005040 con 0x7fb9e80f4880 2023-11-15T04:56:56.618 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.615+0000 7fb9e6d9d700 1 --2- 172.21.15.178:0/796852307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9e80f4880 0x7fb9e80f7470 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fb9dc013510 tx=0x7fb9dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:56.619 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.615+0000 7fb9cf7fe700 1 -- 172.21.15.178:0/796852307 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9dc009320 con 0x7fb9e80f4880 2023-11-15T04:56:56.619 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.615+0000 7fb9cf7fe700 1 -- 172.21.15.178:0/796852307 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9dc013f00 con 0x7fb9e80f4880 2023-11-15T04:56:56.619 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.615+0000 7fb9ed088700 1 -- 172.21.15.178:0/796852307 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e80f5a90 con 0x7fb9e80f4880 2023-11-15T04:56:56.619 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.615+0000 7fb9cf7fe700 1 -- 172.21.15.178:0/796852307 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9dc014dc0 con 0x7fb9e80f4880 2023-11-15T04:56:56.619 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.615+0000 7fb9ed088700 1 -- 172.21.15.178:0/796852307 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e80f5f20 con 0x7fb9e80f4880 2023-11-15T04:56:56.619 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.616+0000 7fb9cf7fe700 1 -- 172.21.15.178:0/796852307 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb9dc014420 con 0x7fb9e80f4880 2023-11-15T04:56:56.620 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.616+0000 7fb9ed088700 1 -- 172.21.15.178:0/796852307 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e8059100 con 0x7fb9e80f4880 2023-11-15T04:56:56.620 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.616+0000 7fb9cf7fe700 1 --2- 172.21.15.178:0/796852307 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb9d002ff60 0x7fb9d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:56.620 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.616+0000 7fb9cf7fe700 1 -- 172.21.15.178:0/796852307 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb9dc041050 con 0x7fb9e80f4880 2023-11-15T04:56:56.620 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.616+0000 7fb9e659c700 1 --2- 172.21.15.178:0/796852307 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb9d002ff60 0x7fb9d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:56.620 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.616+0000 7fb9e659c700 1 --2- 172.21.15.178:0/796852307 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb9d002ff60 0x7fb9d0032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fb9d80096a0 tx=0x7fb9d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:56.621 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.621+0000 7fb9cf7fe700 1 -- 172.21.15.178:0/796852307 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9dc0095f0 con 0x7fb9e80f4880 2023-11-15T04:56:56.825 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.825+0000 7fb9ed088700 1 -- 172.21.15.178:0/796852307 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9e80fbf50 con 0x7fb9e80f4880 2023-11-15T04:56:56.826 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.826+0000 7fb9cf7fe700 1 -- 172.21.15.178:0/796852307 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb9dc01cb10 con 0x7fb9e80f4880 2023-11-15T04:56:56.826 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:56:56.826 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:56:56.828 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.828+0000 7fb9ed088700 1 -- 172.21.15.178:0/796852307 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb9d002ff60 msgr2=0x7fb9d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:56.828 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.828+0000 7fb9ed088700 1 --2- 172.21.15.178:0/796852307 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb9d002ff60 0x7fb9d0032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fb9d80096a0 tx=0x7fb9d8006b40).stop 2023-11-15T04:56:56.829 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.828+0000 7fb9ed088700 1 -- 172.21.15.178:0/796852307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9e80f4880 msgr2=0x7fb9e80f7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:56.829 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.828+0000 7fb9ed088700 1 --2- 172.21.15.178:0/796852307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9e80f4880 0x7fb9e80f7470 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fb9dc013510 tx=0x7fb9dc00b040).stop 2023-11-15T04:56:56.829 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.828+0000 7fb9ed088700 1 -- 172.21.15.178:0/796852307 shutdown_connections 2023-11-15T04:56:56.829 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.828+0000 7fb9ed088700 1 --2- 172.21.15.178:0/796852307 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb9d002ff60 0x7fb9d0032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:56.829 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.828+0000 7fb9ed088700 1 --2- 172.21.15.178:0/796852307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9e80f4880 0x7fb9e80f7470 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:56.830 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.828+0000 7fb9ed088700 1 -- 172.21.15.178:0/796852307 >> 172.21.15.178:0/796852307 conn(0x7fb9e80f02e0 msgr2=0x7fb9e80f0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:56.830 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.828+0000 7fb9ed088700 1 -- 172.21.15.178:0/796852307 shutdown_connections 2023-11-15T04:56:56.830 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:56.828+0000 7fb9ed088700 1 -- 172.21.15.178:0/796852307 wait complete. 2023-11-15T04:56:56.830 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:56:58.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:57 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:56.321981+0000 mgr.smithi163.qesyrx (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:56:58.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:57 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:56.826282+0000 mon.smithi163 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.178:0/796852307' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:56:58.204 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:56:58.205 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:56:59.313 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.310+0000 7f8ec7adf700 1 -- 172.21.15.178:0/1047616209 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ec00f5410 msgr2=0x7f8ec00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:59.313 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.310+0000 7f8ec7adf700 1 --2- 172.21.15.178:0/1047616209 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ec00f5410 0x7f8ec00f5830 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f8eb0009fb0 tx=0x7f8eb0009b30).stop 2023-11-15T04:56:59.313 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.310+0000 7f8ec7adf700 1 -- 172.21.15.178:0/1047616209 shutdown_connections 2023-11-15T04:56:59.314 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.310+0000 7f8ec7adf700 1 --2- 172.21.15.178:0/1047616209 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ec00f5410 0x7f8ec00f5830 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:59.314 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.310+0000 7f8ec7adf700 1 -- 172.21.15.178:0/1047616209 >> 172.21.15.178:0/1047616209 conn(0x7f8ec00f0b70 msgr2=0x7f8ec00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:59.314 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.310+0000 7f8ec7adf700 1 -- 172.21.15.178:0/1047616209 shutdown_connections 2023-11-15T04:56:59.314 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.310+0000 7f8ec7adf700 1 -- 172.21.15.178:0/1047616209 wait complete. 2023-11-15T04:56:59.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.311+0000 7f8ec7adf700 1 Processor -- start 2023-11-15T04:56:59.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.311+0000 7f8ec7adf700 1 -- start start 2023-11-15T04:56:59.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.311+0000 7f8ec7adf700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ec00f5410 0x7f8ec00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:59.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.311+0000 7f8ec7adf700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec00fb820 con 0x7f8ec00f5410 2023-11-15T04:56:59.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.311+0000 7f8ec587b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ec00f5410 0x7f8ec00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:59.316 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.311+0000 7f8ec587b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ec00f5410 0x7f8ec00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:34750/0 (socket says 172.21.15.178:34750) 2023-11-15T04:56:59.316 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.311+0000 7f8ec587b700 1 -- 172.21.15.178:0/773432681 learned_addr learned my addr 172.21.15.178:0/773432681 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:56:59.316 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.312+0000 7f8ec587b700 1 -- 172.21.15.178:0/773432681 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb0005040 con 0x7f8ec00f5410 2023-11-15T04:56:59.316 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.312+0000 7f8ec587b700 1 --2- 172.21.15.178:0/773432681 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ec00f5410 0x7f8ec00fd3a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f8eb0013510 tx=0x7f8eb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:59.317 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.312+0000 7f8eb67fc700 1 -- 172.21.15.178:0/773432681 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8eb0009320 con 0x7f8ec00f5410 2023-11-15T04:56:59.317 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.312+0000 7f8eb67fc700 1 -- 172.21.15.178:0/773432681 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8eb0013f00 con 0x7f8ec00f5410 2023-11-15T04:56:59.317 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.312+0000 7f8ec7adf700 1 -- 172.21.15.178:0/773432681 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec00fb9c0 con 0x7f8ec00f5410 2023-11-15T04:56:59.317 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.312+0000 7f8eb67fc700 1 -- 172.21.15.178:0/773432681 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8eb0014d90 con 0x7f8ec00f5410 2023-11-15T04:56:59.318 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.312+0000 7f8ec7adf700 1 -- 172.21.15.178:0/773432681 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec00fbe50 con 0x7f8ec00f5410 2023-11-15T04:56:59.319 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.313+0000 7f8eb67fc700 1 -- 172.21.15.178:0/773432681 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8eb00095f0 con 0x7f8ec00f5410 2023-11-15T04:56:59.319 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.313+0000 7f8ec7adf700 1 -- 172.21.15.178:0/773432681 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec0059100 con 0x7f8ec00f5410 2023-11-15T04:56:59.319 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.313+0000 7f8eb67fc700 1 --2- 172.21.15.178:0/773432681 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8eac02ffb0 0x7f8eac032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:56:59.320 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.313+0000 7f8eb67fc700 1 -- 172.21.15.178:0/773432681 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8eb0040fe0 con 0x7f8ec00f5410 2023-11-15T04:56:59.320 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.313+0000 7f8ec507a700 1 --2- 172.21.15.178:0/773432681 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8eac02ffb0 0x7f8eac032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:56:59.320 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.314+0000 7f8ec507a700 1 --2- 172.21.15.178:0/773432681 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8eac02ffb0 0x7f8eac032470 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f8ebc0096a0 tx=0x7f8ebc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:56:59.320 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.318+0000 7f8eb67fc700 1 -- 172.21.15.178:0/773432681 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8eb006d2b0 con 0x7f8ec00f5410 2023-11-15T04:56:59.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.522+0000 7f8ec7adf700 1 -- 172.21.15.178:0/773432681 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ec00fae60 con 0x7f8ec00f5410 2023-11-15T04:56:59.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.523+0000 7f8eb67fc700 1 -- 172.21.15.178:0/773432681 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8eb0014420 con 0x7f8ec00f5410 2023-11-15T04:56:59.524 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:56:59.524 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:56:59.525 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.524+0000 7f8ec7adf700 1 -- 172.21.15.178:0/773432681 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8eac02ffb0 msgr2=0x7f8eac032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:59.525 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.524+0000 7f8ec7adf700 1 --2- 172.21.15.178:0/773432681 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8eac02ffb0 0x7f8eac032470 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f8ebc0096a0 tx=0x7f8ebc006b40).stop 2023-11-15T04:56:59.526 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.524+0000 7f8ec7adf700 1 -- 172.21.15.178:0/773432681 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ec00f5410 msgr2=0x7f8ec00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:56:59.526 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.524+0000 7f8ec7adf700 1 --2- 172.21.15.178:0/773432681 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ec00f5410 0x7f8ec00fd3a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f8eb0013510 tx=0x7f8eb000b040).stop 2023-11-15T04:56:59.526 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.524+0000 7f8ec7adf700 1 -- 172.21.15.178:0/773432681 shutdown_connections 2023-11-15T04:56:59.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.524+0000 7f8ec7adf700 1 --2- 172.21.15.178:0/773432681 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8eac02ffb0 0x7f8eac032470 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:59.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.524+0000 7f8ec7adf700 1 --2- 172.21.15.178:0/773432681 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ec00f5410 0x7f8ec00fd3a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:56:59.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.524+0000 7f8ec7adf700 1 -- 172.21.15.178:0/773432681 >> 172.21.15.178:0/773432681 conn(0x7f8ec00f0b70 msgr2=0x7f8ec005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:56:59.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.525+0000 7f8ec7adf700 1 -- 172.21.15.178:0/773432681 shutdown_connections 2023-11-15T04:56:59.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:56:59.525+0000 7f8ec7adf700 1 -- 172.21.15.178:0/773432681 wait complete. 2023-11-15T04:56:59.528 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:00.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:59 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:56:58.322255+0000 mgr.smithi163.qesyrx (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:00.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:56:59 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:56:59.523682+0000 mon.smithi163 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.178:0/773432681' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:00.941 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:00.942 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:57:02.113 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.112+0000 7f299cfee700 1 -- 172.21.15.178:0/994960948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2998060280 msgr2=0x7f29980f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:02.113 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.112+0000 7f299cfee700 1 --2- 172.21.15.178:0/994960948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2998060280 0x7f29980f90f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f298c009fb0 tx=0x7f298c009b30).stop 2023-11-15T04:57:02.113 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.112+0000 7f299cfee700 1 -- 172.21.15.178:0/994960948 shutdown_connections 2023-11-15T04:57:02.114 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.112+0000 7f299cfee700 1 --2- 172.21.15.178:0/994960948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2998060280 0x7f29980f90f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:02.114 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.112+0000 7f299cfee700 1 -- 172.21.15.178:0/994960948 >> 172.21.15.178:0/994960948 conn(0x7f29980f0b70 msgr2=0x7f29980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:02.114 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.112+0000 7f299cfee700 1 -- 172.21.15.178:0/994960948 shutdown_connections 2023-11-15T04:57:02.114 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.112+0000 7f299cfee700 1 -- 172.21.15.178:0/994960948 wait complete. 2023-11-15T04:57:02.116 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.113+0000 7f299cfee700 1 Processor -- start 2023-11-15T04:57:02.117 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.113+0000 7f299cfee700 1 -- start start 2023-11-15T04:57:02.117 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.113+0000 7f299cfee700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2998060280 0x7f29981066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:02.117 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.113+0000 7f299cfee700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2998106bd0 con 0x7f2998060280 2023-11-15T04:57:02.117 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.113+0000 7f299659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2998060280 0x7f29981066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:02.117 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.113+0000 7f299659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2998060280 0x7f29981066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:34760/0 (socket says 172.21.15.178:34760) 2023-11-15T04:57:02.118 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.113+0000 7f299659c700 1 -- 172.21.15.178:0/801778394 learned_addr learned my addr 172.21.15.178:0/801778394 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:57:02.118 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.113+0000 7f299659c700 1 -- 172.21.15.178:0/801778394 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298c005040 con 0x7f2998060280 2023-11-15T04:57:02.118 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.113+0000 7f299659c700 1 --2- 172.21.15.178:0/801778394 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2998060280 0x7f29981066e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f298c013510 tx=0x7f298c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:02.118 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.114+0000 7f2986ffd700 1 -- 172.21.15.178:0/801778394 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f298c009320 con 0x7f2998060280 2023-11-15T04:57:02.119 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.114+0000 7f299cfee700 1 -- 172.21.15.178:0/801778394 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2998106dd0 con 0x7f2998060280 2023-11-15T04:57:02.119 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.114+0000 7f2986ffd700 1 -- 172.21.15.178:0/801778394 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f298c013f00 con 0x7f2998060280 2023-11-15T04:57:02.119 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.114+0000 7f2986ffd700 1 -- 172.21.15.178:0/801778394 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f298c014dc0 con 0x7f2998060280 2023-11-15T04:57:02.119 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.114+0000 7f299cfee700 1 -- 172.21.15.178:0/801778394 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2998107260 con 0x7f2998060280 2023-11-15T04:57:02.119 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.114+0000 7f2986ffd700 1 -- 172.21.15.178:0/801778394 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f298c014420 con 0x7f2998060280 2023-11-15T04:57:02.119 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.114+0000 7f299cfee700 1 -- 172.21.15.178:0/801778394 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2998045ad0 con 0x7f2998060280 2023-11-15T04:57:02.120 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.115+0000 7f2986ffd700 1 --2- 172.21.15.178:0/801778394 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f298002ff60 0x7f2980032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:02.120 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.115+0000 7f2986ffd700 1 -- 172.21.15.178:0/801778394 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f298c041020 con 0x7f2998060280 2023-11-15T04:57:02.120 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.115+0000 7f2995d9b700 1 --2- 172.21.15.178:0/801778394 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f298002ff60 0x7f2980032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:02.120 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.115+0000 7f2995d9b700 1 --2- 172.21.15.178:0/801778394 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f298002ff60 0x7f2980032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f29880096a0 tx=0x7f2988006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:02.121 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.119+0000 7f2986ffd700 1 -- 172.21.15.178:0/801778394 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f298c0095f0 con 0x7f2998060280 2023-11-15T04:57:02.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:01 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:00.322568+0000 mgr.smithi163.qesyrx (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:02.326 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.326+0000 7f299cfee700 1 -- 172.21.15.178:0/801778394 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2998108110 con 0x7f2998060280 2023-11-15T04:57:02.327 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.327+0000 7f2986ffd700 1 -- 172.21.15.178:0/801778394 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f298c01cb10 con 0x7f2998060280 2023-11-15T04:57:02.327 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:57:02.328 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:57:02.329 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.328+0000 7f299cfee700 1 -- 172.21.15.178:0/801778394 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f298002ff60 msgr2=0x7f2980032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:02.329 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.328+0000 7f299cfee700 1 --2- 172.21.15.178:0/801778394 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f298002ff60 0x7f2980032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f29880096a0 tx=0x7f2988006b40).stop 2023-11-15T04:57:02.329 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.328+0000 7f299cfee700 1 -- 172.21.15.178:0/801778394 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2998060280 msgr2=0x7f29981066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:02.330 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.328+0000 7f299cfee700 1 --2- 172.21.15.178:0/801778394 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2998060280 0x7f29981066e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f298c013510 tx=0x7f298c00b040).stop 2023-11-15T04:57:02.330 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.328+0000 7f299cfee700 1 -- 172.21.15.178:0/801778394 shutdown_connections 2023-11-15T04:57:02.330 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.328+0000 7f299cfee700 1 --2- 172.21.15.178:0/801778394 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f298002ff60 0x7f2980032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:02.330 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.328+0000 7f299cfee700 1 --2- 172.21.15.178:0/801778394 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2998060280 0x7f29981066e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:02.330 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.328+0000 7f299cfee700 1 -- 172.21.15.178:0/801778394 >> 172.21.15.178:0/801778394 conn(0x7f29980f0b70 msgr2=0x7f29980f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:02.331 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.328+0000 7f299cfee700 1 -- 172.21.15.178:0/801778394 shutdown_connections 2023-11-15T04:57:02.331 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:02.329+0000 7f299cfee700 1 -- 172.21.15.178:0/801778394 wait complete. 2023-11-15T04:57:02.332 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:03.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:02 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:02.327252+0000 mon.smithi163 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.178:0/801778394' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:03.701 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:03.701 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:57:04.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:03 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:02.322895+0000 mgr.smithi163.qesyrx (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:04.878 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.878+0000 7fdedd3d7700 1 -- 172.21.15.178:0/2493670990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fded80f47c0 msgr2=0x7fded80f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:04.879 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.878+0000 7fdedd3d7700 1 --2- 172.21.15.178:0/2493670990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fded80f47c0 0x7fded80f4be0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fdec8009fb0 tx=0x7fdec8009b30).stop 2023-11-15T04:57:04.879 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.878+0000 7fdedd3d7700 1 -- 172.21.15.178:0/2493670990 shutdown_connections 2023-11-15T04:57:04.879 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.878+0000 7fdedd3d7700 1 --2- 172.21.15.178:0/2493670990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fded80f47c0 0x7fded80f4be0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:04.879 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.878+0000 7fdedd3d7700 1 -- 172.21.15.178:0/2493670990 >> 172.21.15.178:0/2493670990 conn(0x7fded80f02e0 msgr2=0x7fded80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:04.880 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.878+0000 7fdedd3d7700 1 -- 172.21.15.178:0/2493670990 shutdown_connections 2023-11-15T04:57:04.880 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.878+0000 7fdedd3d7700 1 -- 172.21.15.178:0/2493670990 wait complete. 2023-11-15T04:57:04.882 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.878+0000 7fdedd3d7700 1 Processor -- start 2023-11-15T04:57:04.882 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.878+0000 7fdedd3d7700 1 -- start start 2023-11-15T04:57:04.882 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.879+0000 7fdedd3d7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fded80f47c0 0x7fded8106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:04.882 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.879+0000 7fdedd3d7700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded8106bf0 con 0x7fded80f47c0 2023-11-15T04:57:04.883 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.879+0000 7fded6ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fded80f47c0 0x7fded8106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:04.883 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.879+0000 7fded6ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fded80f47c0 0x7fded8106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:37964/0 (socket says 172.21.15.178:37964) 2023-11-15T04:57:04.883 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.879+0000 7fded6ffd700 1 -- 172.21.15.178:0/3249739005 learned_addr learned my addr 172.21.15.178:0/3249739005 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:57:04.883 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.879+0000 7fded6ffd700 1 -- 172.21.15.178:0/3249739005 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec8005040 con 0x7fded80f47c0 2023-11-15T04:57:04.883 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.879+0000 7fded6ffd700 1 --2- 172.21.15.178:0/3249739005 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fded80f47c0 0x7fded8106700 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fdec8013510 tx=0x7fdec800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:04.884 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.879+0000 7fdebf7fe700 1 -- 172.21.15.178:0/3249739005 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdec8009320 con 0x7fded80f47c0 2023-11-15T04:57:04.884 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.879+0000 7fdedd3d7700 1 -- 172.21.15.178:0/3249739005 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded8106df0 con 0x7fded80f47c0 2023-11-15T04:57:04.884 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.879+0000 7fdebf7fe700 1 -- 172.21.15.178:0/3249739005 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdec8013f00 con 0x7fded80f47c0 2023-11-15T04:57:04.884 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.879+0000 7fdebf7fe700 1 -- 172.21.15.178:0/3249739005 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdec8014d90 con 0x7fded80f47c0 2023-11-15T04:57:04.884 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.879+0000 7fdedd3d7700 1 -- 172.21.15.178:0/3249739005 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fded8107280 con 0x7fded80f47c0 2023-11-15T04:57:04.885 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.880+0000 7fdebf7fe700 1 -- 172.21.15.178:0/3249739005 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdec8014420 con 0x7fded80f47c0 2023-11-15T04:57:04.885 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.880+0000 7fdedd3d7700 1 -- 172.21.15.178:0/3249739005 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fded8045ad0 con 0x7fded80f47c0 2023-11-15T04:57:04.885 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.880+0000 7fdebf7fe700 1 --2- 172.21.15.178:0/3249739005 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdec002ff60 0x7fdec0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:04.885 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.880+0000 7fdebf7fe700 1 -- 172.21.15.178:0/3249739005 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdec8041000 con 0x7fded80f47c0 2023-11-15T04:57:04.886 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.881+0000 7fded67fc700 1 --2- 172.21.15.178:0/3249739005 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdec002ff60 0x7fdec0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:04.886 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.881+0000 7fded67fc700 1 --2- 172.21.15.178:0/3249739005 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdec002ff60 0x7fdec0032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fdecc0096a0 tx=0x7fdecc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:04.886 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:04.885+0000 7fdebf7fe700 1 -- 172.21.15.178:0/3249739005 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdec80095f0 con 0x7fded80f47c0 2023-11-15T04:57:05.091 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:05.091+0000 7fdedd3d7700 1 -- 172.21.15.178:0/3249739005 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fded8108130 con 0x7fded80f47c0 2023-11-15T04:57:05.092 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:05.092+0000 7fdebf7fe700 1 -- 172.21.15.178:0/3249739005 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdec801db70 con 0x7fded80f47c0 2023-11-15T04:57:05.092 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:57:05.092 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:57:05.093 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:05.093+0000 7fdedd3d7700 1 -- 172.21.15.178:0/3249739005 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdec002ff60 msgr2=0x7fdec0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:05.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:05.093+0000 7fdedd3d7700 1 --2- 172.21.15.178:0/3249739005 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdec002ff60 0x7fdec0032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fdecc0096a0 tx=0x7fdecc006b40).stop 2023-11-15T04:57:05.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:05.093+0000 7fdedd3d7700 1 -- 172.21.15.178:0/3249739005 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fded80f47c0 msgr2=0x7fded8106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:05.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:05.093+0000 7fdedd3d7700 1 --2- 172.21.15.178:0/3249739005 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fded80f47c0 0x7fded8106700 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fdec8013510 tx=0x7fdec800b040).stop 2023-11-15T04:57:05.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:05.093+0000 7fdedd3d7700 1 -- 172.21.15.178:0/3249739005 shutdown_connections 2023-11-15T04:57:05.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:05.093+0000 7fdedd3d7700 1 --2- 172.21.15.178:0/3249739005 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdec002ff60 0x7fdec0032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:05.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:05.093+0000 7fdedd3d7700 1 --2- 172.21.15.178:0/3249739005 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fded80f47c0 0x7fded8106700 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:05.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:05.093+0000 7fdedd3d7700 1 -- 172.21.15.178:0/3249739005 >> 172.21.15.178:0/3249739005 conn(0x7fded80f02e0 msgr2=0x7fded80f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:05.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:05.093+0000 7fdedd3d7700 1 -- 172.21.15.178:0/3249739005 shutdown_connections 2023-11-15T04:57:05.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:05.093+0000 7fdedd3d7700 1 -- 172.21.15.178:0/3249739005 wait complete. 2023-11-15T04:57:05.097 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:06.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:05 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:04.323229+0000 mgr.smithi163.qesyrx (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:06.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:05 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:05.092302+0000 mon.smithi163 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.178:0/3249739005' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:06.471 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:06.472 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:57:07.586 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.585+0000 7f4ed1b92700 1 -- 172.21.15.178:0/1670189127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ecc0f5410 msgr2=0x7f4ecc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:07.587 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.585+0000 7f4ed1b92700 1 --2- 172.21.15.178:0/1670189127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ecc0f5410 0x7f4ecc0f5830 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f4ec0009fb0 tx=0x7f4ec0009b30).stop 2023-11-15T04:57:07.587 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.585+0000 7f4ed1b92700 1 -- 172.21.15.178:0/1670189127 shutdown_connections 2023-11-15T04:57:07.587 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.585+0000 7f4ed1b92700 1 --2- 172.21.15.178:0/1670189127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ecc0f5410 0x7f4ecc0f5830 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:07.587 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.585+0000 7f4ed1b92700 1 -- 172.21.15.178:0/1670189127 >> 172.21.15.178:0/1670189127 conn(0x7f4ecc0f0b70 msgr2=0x7f4ecc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:07.588 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.585+0000 7f4ed1b92700 1 -- 172.21.15.178:0/1670189127 shutdown_connections 2023-11-15T04:57:07.588 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.585+0000 7f4ed1b92700 1 -- 172.21.15.178:0/1670189127 wait complete. 2023-11-15T04:57:07.588 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.585+0000 7f4ed1b92700 1 Processor -- start 2023-11-15T04:57:07.588 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.586+0000 7f4ed1b92700 1 -- start start 2023-11-15T04:57:07.588 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.586+0000 7f4ed1b92700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ecc0f5410 0x7f4ecc109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:07.588 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.586+0000 7f4ed1b92700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ecc060220 con 0x7f4ecc0f5410 2023-11-15T04:57:07.589 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.586+0000 7f4ecb7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ecc0f5410 0x7f4ecc109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:07.589 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.586+0000 7f4ecb7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ecc0f5410 0x7f4ecc109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:37968/0 (socket says 172.21.15.178:37968) 2023-11-15T04:57:07.589 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.586+0000 7f4ecb7fe700 1 -- 172.21.15.178:0/4128034002 learned_addr learned my addr 172.21.15.178:0/4128034002 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:57:07.589 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.586+0000 7f4ecb7fe700 1 -- 172.21.15.178:0/4128034002 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec0005040 con 0x7f4ecc0f5410 2023-11-15T04:57:07.589 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.586+0000 7f4ecb7fe700 1 --2- 172.21.15.178:0/4128034002 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ecc0f5410 0x7f4ecc109450 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f4ec0014000 tx=0x7f4ec000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:07.591 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.587+0000 7f4eb3fff700 1 -- 172.21.15.178:0/4128034002 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ec00095f0 con 0x7f4ecc0f5410 2023-11-15T04:57:07.591 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.587+0000 7f4ed1b92700 1 -- 172.21.15.178:0/4128034002 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ecc060420 con 0x7f4ecc0f5410 2023-11-15T04:57:07.591 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.587+0000 7f4eb3fff700 1 -- 172.21.15.178:0/4128034002 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ec0007e70 con 0x7f4ecc0f5410 2023-11-15T04:57:07.591 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.587+0000 7f4eb3fff700 1 -- 172.21.15.178:0/4128034002 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ec0014ea0 con 0x7f4ecc0f5410 2023-11-15T04:57:07.592 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.587+0000 7f4ed1b92700 1 -- 172.21.15.178:0/4128034002 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ecc109b00 con 0x7f4ecc0f5410 2023-11-15T04:57:07.592 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.587+0000 7f4eb3fff700 1 -- 172.21.15.178:0/4128034002 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4ec001d4e0 con 0x7f4ecc0f5410 2023-11-15T04:57:07.592 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.587+0000 7f4ed1b92700 1 -- 172.21.15.178:0/4128034002 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ecc059100 con 0x7f4ecc0f5410 2023-11-15T04:57:07.592 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.587+0000 7f4eb3fff700 1 --2- 172.21.15.178:0/4128034002 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4eb402ff60 0x7f4eb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:07.592 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.588+0000 7f4eb3fff700 1 -- 172.21.15.178:0/4128034002 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ec0040bd0 con 0x7f4ecc0f5410 2023-11-15T04:57:07.593 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.588+0000 7f4ecaffd700 1 --2- 172.21.15.178:0/4128034002 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4eb402ff60 0x7f4eb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:07.593 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.588+0000 7f4ecaffd700 1 --2- 172.21.15.178:0/4128034002 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4eb402ff60 0x7f4eb4032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f4ebc0096a0 tx=0x7f4ebc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:07.593 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.592+0000 7f4eb3fff700 1 -- 172.21.15.178:0/4128034002 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ec0025ae0 con 0x7f4ecc0f5410 2023-11-15T04:57:07.799 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.799+0000 7f4ed1b92700 1 -- 172.21.15.178:0/4128034002 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ecc045ad0 con 0x7f4ecc0f5410 2023-11-15T04:57:07.800 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.800+0000 7f4eb3fff700 1 -- 172.21.15.178:0/4128034002 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4ec0025ae0 con 0x7f4ecc0f5410 2023-11-15T04:57:07.800 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:57:07.801 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:57:07.802 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.801+0000 7f4ed1b92700 1 -- 172.21.15.178:0/4128034002 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4eb402ff60 msgr2=0x7f4eb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:07.802 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.801+0000 7f4ed1b92700 1 --2- 172.21.15.178:0/4128034002 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4eb402ff60 0x7f4eb4032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f4ebc0096a0 tx=0x7f4ebc006b40).stop 2023-11-15T04:57:07.802 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.801+0000 7f4ed1b92700 1 -- 172.21.15.178:0/4128034002 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ecc0f5410 msgr2=0x7f4ecc109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:07.802 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.801+0000 7f4ed1b92700 1 --2- 172.21.15.178:0/4128034002 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ecc0f5410 0x7f4ecc109450 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f4ec0014000 tx=0x7f4ec000b040).stop 2023-11-15T04:57:07.803 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.801+0000 7f4ed1b92700 1 -- 172.21.15.178:0/4128034002 shutdown_connections 2023-11-15T04:57:07.803 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.801+0000 7f4ed1b92700 1 --2- 172.21.15.178:0/4128034002 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4eb402ff60 0x7f4eb4032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:07.803 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.801+0000 7f4ed1b92700 1 --2- 172.21.15.178:0/4128034002 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ecc0f5410 0x7f4ecc109450 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:07.803 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.801+0000 7f4ed1b92700 1 -- 172.21.15.178:0/4128034002 >> 172.21.15.178:0/4128034002 conn(0x7f4ecc0f0b70 msgr2=0x7f4ecc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:07.804 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.801+0000 7f4ed1b92700 1 -- 172.21.15.178:0/4128034002 shutdown_connections 2023-11-15T04:57:07.804 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:07.801+0000 7f4ed1b92700 1 -- 172.21.15.178:0/4128034002 wait complete. 2023-11-15T04:57:07.804 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:08.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:07 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:06.323588+0000 mgr.smithi163.qesyrx (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:09.127 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:09.128 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:57:09.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:08 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:07.800273+0000 mon.smithi163 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.178:0/4128034002' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:10.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:09 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:08.323943+0000 mgr.smithi163.qesyrx (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:10.292 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.290+0000 7f6008b6a700 1 -- 172.21.15.178:0/474199678 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60040f6510 msgr2=0x7f60040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:10.293 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.290+0000 7f6008b6a700 1 --2- 172.21.15.178:0/474199678 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60040f6510 0x7f60040f6930 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f5ff4009fb0 tx=0x7f5ff4009b30).stop 2023-11-15T04:57:10.293 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.290+0000 7f6008b6a700 1 -- 172.21.15.178:0/474199678 shutdown_connections 2023-11-15T04:57:10.293 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.290+0000 7f6008b6a700 1 --2- 172.21.15.178:0/474199678 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60040f6510 0x7f60040f6930 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:10.294 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.290+0000 7f6008b6a700 1 -- 172.21.15.178:0/474199678 >> 172.21.15.178:0/474199678 conn(0x7f60040f1c70 msgr2=0x7f60040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:10.294 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.290+0000 7f6008b6a700 1 -- 172.21.15.178:0/474199678 shutdown_connections 2023-11-15T04:57:10.294 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.290+0000 7f6008b6a700 1 -- 172.21.15.178:0/474199678 wait complete. 2023-11-15T04:57:10.294 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.291+0000 7f6008b6a700 1 Processor -- start 2023-11-15T04:57:10.295 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.291+0000 7f6008b6a700 1 -- start start 2023-11-15T04:57:10.295 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.291+0000 7f6008b6a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60040f6510 0x7f600406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:10.295 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.291+0000 7f6008b6a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6004069e20 con 0x7f60040f6510 2023-11-15T04:57:10.296 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.291+0000 7f600259c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60040f6510 0x7f600406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:10.296 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.291+0000 7f600259c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60040f6510 0x7f600406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:37996/0 (socket says 172.21.15.178:37996) 2023-11-15T04:57:10.296 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.291+0000 7f600259c700 1 -- 172.21.15.178:0/4073847271 learned_addr learned my addr 172.21.15.178:0/4073847271 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:57:10.296 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.291+0000 7f600259c700 1 -- 172.21.15.178:0/4073847271 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff4005040 con 0x7f60040f6510 2023-11-15T04:57:10.297 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.291+0000 7f600259c700 1 --2- 172.21.15.178:0/4073847271 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60040f6510 0x7f600406b9a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f5ff4013510 tx=0x7f5ff400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:10.297 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.292+0000 7f5ffaffd700 1 -- 172.21.15.178:0/4073847271 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ff4009320 con 0x7f60040f6510 2023-11-15T04:57:10.297 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.292+0000 7f6008b6a700 1 -- 172.21.15.178:0/4073847271 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6004069fc0 con 0x7f60040f6510 2023-11-15T04:57:10.297 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.292+0000 7f5ffaffd700 1 -- 172.21.15.178:0/4073847271 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ff4013f00 con 0x7f60040f6510 2023-11-15T04:57:10.298 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.292+0000 7f5ffaffd700 1 -- 172.21.15.178:0/4073847271 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ff4014d90 con 0x7f60040f6510 2023-11-15T04:57:10.298 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.292+0000 7f6008b6a700 1 -- 172.21.15.178:0/4073847271 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f600406a450 con 0x7f60040f6510 2023-11-15T04:57:10.299 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.292+0000 7f5ffaffd700 1 -- 172.21.15.178:0/4073847271 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ff4014420 con 0x7f60040f6510 2023-11-15T04:57:10.299 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.292+0000 7f6008b6a700 1 -- 172.21.15.178:0/4073847271 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6004059100 con 0x7f60040f6510 2023-11-15T04:57:10.300 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.293+0000 7f5ffaffd700 1 --2- 172.21.15.178:0/4073847271 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5ff002ff60 0x7f5ff0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:10.300 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.293+0000 7f5ffaffd700 1 -- 172.21.15.178:0/4073847271 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ff4041000 con 0x7f60040f6510 2023-11-15T04:57:10.300 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.293+0000 7f6001d9b700 1 --2- 172.21.15.178:0/4073847271 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5ff002ff60 0x7f5ff0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:10.300 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.293+0000 7f6001d9b700 1 --2- 172.21.15.178:0/4073847271 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5ff002ff60 0x7f5ff0032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f5fec0096a0 tx=0x7f5fec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:10.300 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.297+0000 7f5ffaffd700 1 -- 172.21.15.178:0/4073847271 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ff40095f0 con 0x7f60040f6510 2023-11-15T04:57:10.505 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.505+0000 7f6008b6a700 1 -- 172.21.15.178:0/4073847271 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60040fbf60 con 0x7f60040f6510 2023-11-15T04:57:10.506 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.505+0000 7f5ffaffd700 1 -- 172.21.15.178:0/4073847271 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5ff401db70 con 0x7f60040f6510 2023-11-15T04:57:10.507 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:57:10.507 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:57:10.508 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.507+0000 7f6008b6a700 1 -- 172.21.15.178:0/4073847271 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5ff002ff60 msgr2=0x7f5ff0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:10.509 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.507+0000 7f6008b6a700 1 --2- 172.21.15.178:0/4073847271 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5ff002ff60 0x7f5ff0032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f5fec0096a0 tx=0x7f5fec006b40).stop 2023-11-15T04:57:10.509 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.507+0000 7f6008b6a700 1 -- 172.21.15.178:0/4073847271 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60040f6510 msgr2=0x7f600406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:10.509 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.507+0000 7f6008b6a700 1 --2- 172.21.15.178:0/4073847271 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60040f6510 0x7f600406b9a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f5ff4013510 tx=0x7f5ff400b040).stop 2023-11-15T04:57:10.509 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.507+0000 7f6008b6a700 1 -- 172.21.15.178:0/4073847271 shutdown_connections 2023-11-15T04:57:10.509 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.507+0000 7f6008b6a700 1 --2- 172.21.15.178:0/4073847271 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5ff002ff60 0x7f5ff0032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:10.510 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.507+0000 7f6008b6a700 1 --2- 172.21.15.178:0/4073847271 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60040f6510 0x7f600406b9a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:10.510 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.507+0000 7f6008b6a700 1 -- 172.21.15.178:0/4073847271 >> 172.21.15.178:0/4073847271 conn(0x7f60040f1c70 msgr2=0x7f60040f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:10.510 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.507+0000 7f6008b6a700 1 -- 172.21.15.178:0/4073847271 shutdown_connections 2023-11-15T04:57:10.510 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:10.507+0000 7f6008b6a700 1 -- 172.21.15.178:0/4073847271 wait complete. 2023-11-15T04:57:10.510 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:11.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:10 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:10.506134+0000 mon.smithi163 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.178:0/4073847271' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:11.889 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:11.890 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:57:12.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:11 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:10.324330+0000 mgr.smithi163.qesyrx (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:13.086 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.085+0000 7f187920b700 1 -- 172.21.15.178:0/1439269521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18740f5410 msgr2=0x7f18740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:13.087 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.085+0000 7f187920b700 1 --2- 172.21.15.178:0/1439269521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18740f5410 0x7f18740f5830 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f1868009fb0 tx=0x7f1868009b30).stop 2023-11-15T04:57:13.087 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.085+0000 7f187920b700 1 -- 172.21.15.178:0/1439269521 shutdown_connections 2023-11-15T04:57:13.087 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.085+0000 7f187920b700 1 --2- 172.21.15.178:0/1439269521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18740f5410 0x7f18740f5830 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:13.087 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.085+0000 7f187920b700 1 -- 172.21.15.178:0/1439269521 >> 172.21.15.178:0/1439269521 conn(0x7f18740f0b70 msgr2=0x7f18740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:13.088 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.085+0000 7f187920b700 1 -- 172.21.15.178:0/1439269521 shutdown_connections 2023-11-15T04:57:13.088 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.085+0000 7f187920b700 1 -- 172.21.15.178:0/1439269521 wait complete. 2023-11-15T04:57:13.088 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.086+0000 7f187920b700 1 Processor -- start 2023-11-15T04:57:13.088 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.086+0000 7f187920b700 1 -- start start 2023-11-15T04:57:13.089 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.086+0000 7f187920b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18740f5410 0x7f1874106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:13.089 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.086+0000 7f187920b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1874106c10 con 0x7f18740f5410 2023-11-15T04:57:13.089 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.086+0000 7f1872d9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18740f5410 0x7f1874106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:13.089 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.086+0000 7f1872d9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18740f5410 0x7f1874106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:38018/0 (socket says 172.21.15.178:38018) 2023-11-15T04:57:13.089 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.086+0000 7f1872d9d700 1 -- 172.21.15.178:0/3927828338 learned_addr learned my addr 172.21.15.178:0/3927828338 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:57:13.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.086+0000 7f1872d9d700 1 -- 172.21.15.178:0/3927828338 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1868005040 con 0x7f18740f5410 2023-11-15T04:57:13.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.086+0000 7f1872d9d700 1 --2- 172.21.15.178:0/3927828338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18740f5410 0x7f1874106720 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f1868013510 tx=0x7f186800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:13.091 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.087+0000 7f185b7fe700 1 -- 172.21.15.178:0/3927828338 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1868009320 con 0x7f18740f5410 2023-11-15T04:57:13.091 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.087+0000 7f187920b700 1 -- 172.21.15.178:0/3927828338 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1874106e10 con 0x7f18740f5410 2023-11-15T04:57:13.092 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.087+0000 7f185b7fe700 1 -- 172.21.15.178:0/3927828338 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1868013f00 con 0x7f18740f5410 2023-11-15T04:57:13.092 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.087+0000 7f185b7fe700 1 -- 172.21.15.178:0/3927828338 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1868014d90 con 0x7f18740f5410 2023-11-15T04:57:13.092 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.087+0000 7f187920b700 1 -- 172.21.15.178:0/3927828338 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18741072a0 con 0x7f18740f5410 2023-11-15T04:57:13.093 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.087+0000 7f185b7fe700 1 -- 172.21.15.178:0/3927828338 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f18680095f0 con 0x7f18740f5410 2023-11-15T04:57:13.093 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.087+0000 7f187920b700 1 -- 172.21.15.178:0/3927828338 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1874045ad0 con 0x7f18740f5410 2023-11-15T04:57:13.093 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.088+0000 7f185b7fe700 1 --2- 172.21.15.178:0/3927828338 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f185c02ff60 0x7f185c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:13.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.088+0000 7f185b7fe700 1 -- 172.21.15.178:0/3927828338 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1868040fe0 con 0x7f18740f5410 2023-11-15T04:57:13.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.088+0000 7f187259c700 1 --2- 172.21.15.178:0/3927828338 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f185c02ff60 0x7f185c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:13.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.088+0000 7f187259c700 1 --2- 172.21.15.178:0/3927828338 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f185c02ff60 0x7f185c032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f18640096a0 tx=0x7f1864006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:13.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.092+0000 7f185b7fe700 1 -- 172.21.15.178:0/3927828338 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f186801d7f0 con 0x7f18740f5410 2023-11-15T04:57:13.297 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.297+0000 7f187920b700 1 -- 172.21.15.178:0/3927828338 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1874108150 con 0x7f18740f5410 2023-11-15T04:57:13.301 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.298+0000 7f185b7fe700 1 -- 172.21.15.178:0/3927828338 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1868010230 con 0x7f18740f5410 2023-11-15T04:57:13.302 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:57:13.302 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:57:13.303 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.299+0000 7f187920b700 1 -- 172.21.15.178:0/3927828338 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f185c02ff60 msgr2=0x7f185c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:13.303 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.299+0000 7f187920b700 1 --2- 172.21.15.178:0/3927828338 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f185c02ff60 0x7f185c032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f18640096a0 tx=0x7f1864006b40).stop 2023-11-15T04:57:13.303 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.299+0000 7f187920b700 1 -- 172.21.15.178:0/3927828338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18740f5410 msgr2=0x7f1874106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:13.304 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.299+0000 7f187920b700 1 --2- 172.21.15.178:0/3927828338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18740f5410 0x7f1874106720 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f1868013510 tx=0x7f186800b040).stop 2023-11-15T04:57:13.304 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.299+0000 7f187920b700 1 -- 172.21.15.178:0/3927828338 shutdown_connections 2023-11-15T04:57:13.304 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.299+0000 7f187920b700 1 --2- 172.21.15.178:0/3927828338 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f185c02ff60 0x7f185c032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:13.304 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.299+0000 7f187920b700 1 --2- 172.21.15.178:0/3927828338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18740f5410 0x7f1874106720 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:13.305 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.299+0000 7f187920b700 1 -- 172.21.15.178:0/3927828338 >> 172.21.15.178:0/3927828338 conn(0x7f18740f0b70 msgr2=0x7f187405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:13.305 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.299+0000 7f187920b700 1 -- 172.21.15.178:0/3927828338 shutdown_connections 2023-11-15T04:57:13.306 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:13.299+0000 7f187920b700 1 -- 172.21.15.178:0/3927828338 wait complete. 2023-11-15T04:57:13.306 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:14.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:13 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:12.324630+0000 mgr.smithi163.qesyrx (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:14.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:13 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:13.298355+0000 mon.smithi163 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.178:0/3927828338' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:14.663 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:14.664 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:57:15.886 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.885+0000 7fe4851ed700 1 -- 172.21.15.178:0/631160503 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe480060220 msgr2=0x7fe4800f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:15.886 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.885+0000 7fe4851ed700 1 --2- 172.21.15.178:0/631160503 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe480060220 0x7fe4800f9150 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fe470009fb0 tx=0x7fe470009b30).stop 2023-11-15T04:57:15.887 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.885+0000 7fe4851ed700 1 -- 172.21.15.178:0/631160503 shutdown_connections 2023-11-15T04:57:15.887 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.885+0000 7fe4851ed700 1 --2- 172.21.15.178:0/631160503 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe480060220 0x7fe4800f9150 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:15.887 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.885+0000 7fe4851ed700 1 -- 172.21.15.178:0/631160503 >> 172.21.15.178:0/631160503 conn(0x7fe4800f0b70 msgr2=0x7fe4800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:15.887 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.886+0000 7fe4851ed700 1 -- 172.21.15.178:0/631160503 shutdown_connections 2023-11-15T04:57:15.887 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.886+0000 7fe4851ed700 1 -- 172.21.15.178:0/631160503 wait complete. 2023-11-15T04:57:15.889 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.886+0000 7fe4851ed700 1 Processor -- start 2023-11-15T04:57:15.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.886+0000 7fe4851ed700 1 -- start start 2023-11-15T04:57:15.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.886+0000 7fe4851ed700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe480060220 0x7fe4801066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:15.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.886+0000 7fe4851ed700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe480106ba0 con 0x7fe480060220 2023-11-15T04:57:15.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.887+0000 7fe47ed9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe480060220 0x7fe4801066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:15.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.887+0000 7fe47ed9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe480060220 0x7fe4801066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:39158/0 (socket says 172.21.15.178:39158) 2023-11-15T04:57:15.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.887+0000 7fe47ed9d700 1 -- 172.21.15.178:0/3157850910 learned_addr learned my addr 172.21.15.178:0/3157850910 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:57:15.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.887+0000 7fe47ed9d700 1 -- 172.21.15.178:0/3157850910 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe470005040 con 0x7fe480060220 2023-11-15T04:57:15.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.887+0000 7fe47ed9d700 1 --2- 172.21.15.178:0/3157850910 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe480060220 0x7fe4801066b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fe470013510 tx=0x7fe47000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:15.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.887+0000 7fe4677fe700 1 -- 172.21.15.178:0/3157850910 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe470009320 con 0x7fe480060220 2023-11-15T04:57:15.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.887+0000 7fe4851ed700 1 -- 172.21.15.178:0/3157850910 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe480106da0 con 0x7fe480060220 2023-11-15T04:57:15.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.887+0000 7fe4677fe700 1 -- 172.21.15.178:0/3157850910 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe470013f00 con 0x7fe480060220 2023-11-15T04:57:15.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.887+0000 7fe4677fe700 1 -- 172.21.15.178:0/3157850910 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe470014dc0 con 0x7fe480060220 2023-11-15T04:57:15.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.887+0000 7fe4851ed700 1 -- 172.21.15.178:0/3157850910 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe480107230 con 0x7fe480060220 2023-11-15T04:57:15.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.888+0000 7fe4677fe700 1 -- 172.21.15.178:0/3157850910 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe4700095f0 con 0x7fe480060220 2023-11-15T04:57:15.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.888+0000 7fe4851ed700 1 -- 172.21.15.178:0/3157850910 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe480045ad0 con 0x7fe480060220 2023-11-15T04:57:15.893 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.888+0000 7fe4677fe700 1 --2- 172.21.15.178:0/3157850910 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe46802ff60 0x7fe468032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:15.893 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.888+0000 7fe4677fe700 1 -- 172.21.15.178:0/3157850910 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe4700410c0 con 0x7fe480060220 2023-11-15T04:57:15.893 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.888+0000 7fe47e59c700 1 --2- 172.21.15.178:0/3157850910 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe46802ff60 0x7fe468032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:15.893 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.889+0000 7fe47e59c700 1 --2- 172.21.15.178:0/3157850910 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe46802ff60 0x7fe468032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fe4740096a0 tx=0x7fe474006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:15.894 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:15.893+0000 7fe4677fe700 1 -- 172.21.15.178:0/3157850910 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe470014420 con 0x7fe480060220 2023-11-15T04:57:16.099 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:16.098+0000 7fe4851ed700 1 -- 172.21.15.178:0/3157850910 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe4801080e0 con 0x7fe480060220 2023-11-15T04:57:16.099 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:16.099+0000 7fe4677fe700 1 -- 172.21.15.178:0/3157850910 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe470010230 con 0x7fe480060220 2023-11-15T04:57:16.100 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:57:16.100 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:57:16.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:16.101+0000 7fe4851ed700 1 -- 172.21.15.178:0/3157850910 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe46802ff60 msgr2=0x7fe468032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:16.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:16.101+0000 7fe4851ed700 1 --2- 172.21.15.178:0/3157850910 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe46802ff60 0x7fe468032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fe4740096a0 tx=0x7fe474006b40).stop 2023-11-15T04:57:16.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:16.101+0000 7fe4851ed700 1 -- 172.21.15.178:0/3157850910 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe480060220 msgr2=0x7fe4801066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:16.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:16.101+0000 7fe4851ed700 1 --2- 172.21.15.178:0/3157850910 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe480060220 0x7fe4801066b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fe470013510 tx=0x7fe47000b040).stop 2023-11-15T04:57:16.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:16.101+0000 7fe4851ed700 1 -- 172.21.15.178:0/3157850910 shutdown_connections 2023-11-15T04:57:16.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:16.101+0000 7fe4851ed700 1 --2- 172.21.15.178:0/3157850910 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe46802ff60 0x7fe468032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:16.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:16.101+0000 7fe4851ed700 1 --2- 172.21.15.178:0/3157850910 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe480060220 0x7fe4801066b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:16.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:16.101+0000 7fe4851ed700 1 -- 172.21.15.178:0/3157850910 >> 172.21.15.178:0/3157850910 conn(0x7fe4800f0b70 msgr2=0x7fe4800f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:16.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:16.101+0000 7fe4851ed700 1 -- 172.21.15.178:0/3157850910 shutdown_connections 2023-11-15T04:57:16.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:16.101+0000 7fe4851ed700 1 -- 172.21.15.178:0/3157850910 wait complete. 2023-11-15T04:57:16.103 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:16.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:15 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:14.324893+0000 mgr.smithi163.qesyrx (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:17.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:16.099763+0000 mon.smithi163 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.178:0/3157850910' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:17.491 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:17.492 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:57:18.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:17 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:16.325233+0000 mgr.smithi163.qesyrx (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:18.732 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.730+0000 7f13a0b96700 1 -- 172.21.15.178:0/3076830341 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f139c0601e0 msgr2=0x7f139c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:18.732 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.730+0000 7f13a0b96700 1 --2- 172.21.15.178:0/3076830341 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f139c0601e0 0x7f139c0f8f20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f138c009fb0 tx=0x7f138c009b30).stop 2023-11-15T04:57:18.733 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.730+0000 7f13a0b96700 1 -- 172.21.15.178:0/3076830341 shutdown_connections 2023-11-15T04:57:18.733 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.730+0000 7f13a0b96700 1 --2- 172.21.15.178:0/3076830341 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f139c0601e0 0x7f139c0f8f20 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:18.733 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.730+0000 7f13a0b96700 1 -- 172.21.15.178:0/3076830341 >> 172.21.15.178:0/3076830341 conn(0x7f139c0f09a0 msgr2=0x7f139c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:18.733 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.730+0000 7f13a0b96700 1 -- 172.21.15.178:0/3076830341 shutdown_connections 2023-11-15T04:57:18.733 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.730+0000 7f13a0b96700 1 -- 172.21.15.178:0/3076830341 wait complete. 2023-11-15T04:57:18.733 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.730+0000 7f13a0b96700 1 Processor -- start 2023-11-15T04:57:18.734 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.731+0000 7f13a0b96700 1 -- start start 2023-11-15T04:57:18.734 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.731+0000 7f13a0b96700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f139c0601e0 0x7f139c0f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:18.734 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.731+0000 7f13a0b96700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139c0f4e20 con 0x7f139c0601e0 2023-11-15T04:57:18.736 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.731+0000 7f139b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f139c0601e0 0x7f139c0f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:18.736 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.731+0000 7f139b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f139c0601e0 0x7f139c0f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:39174/0 (socket says 172.21.15.178:39174) 2023-11-15T04:57:18.736 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.731+0000 7f139b7fe700 1 -- 172.21.15.178:0/929517272 learned_addr learned my addr 172.21.15.178:0/929517272 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:57:18.736 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.731+0000 7f139b7fe700 1 -- 172.21.15.178:0/929517272 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c005040 con 0x7f139c0601e0 2023-11-15T04:57:18.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.732+0000 7f139b7fe700 1 --2- 172.21.15.178:0/929517272 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f139c0601e0 0x7f139c0f69a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f138c014000 tx=0x7f138c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:18.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.732+0000 7f1383fff700 1 -- 172.21.15.178:0/929517272 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f138c0095f0 con 0x7f139c0601e0 2023-11-15T04:57:18.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.732+0000 7f13a0b96700 1 -- 172.21.15.178:0/929517272 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f139c0f4fc0 con 0x7f139c0601e0 2023-11-15T04:57:18.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.732+0000 7f1383fff700 1 -- 172.21.15.178:0/929517272 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f138c007e70 con 0x7f139c0601e0 2023-11-15T04:57:18.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.732+0000 7f1383fff700 1 -- 172.21.15.178:0/929517272 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f138c014ea0 con 0x7f139c0601e0 2023-11-15T04:57:18.738 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.732+0000 7f13a0b96700 1 -- 172.21.15.178:0/929517272 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f139c0f5450 con 0x7f139c0601e0 2023-11-15T04:57:18.738 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.733+0000 7f1383fff700 1 -- 172.21.15.178:0/929517272 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f138c025070 con 0x7f139c0601e0 2023-11-15T04:57:18.738 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.733+0000 7f13a0b96700 1 -- 172.21.15.178:0/929517272 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f139c059100 con 0x7f139c0601e0 2023-11-15T04:57:18.738 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.733+0000 7f1383fff700 1 --2- 172.21.15.178:0/929517272 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f138402ff60 0x7f1384032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:18.738 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.733+0000 7f1383fff700 1 -- 172.21.15.178:0/929517272 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f138c0409f0 con 0x7f139c0601e0 2023-11-15T04:57:18.739 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.733+0000 7f139affd700 1 --2- 172.21.15.178:0/929517272 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f138402ff60 0x7f1384032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:18.739 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.734+0000 7f139affd700 1 --2- 172.21.15.178:0/929517272 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f138402ff60 0x7f1384032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f13900096a0 tx=0x7f1390006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:18.739 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.738+0000 7f1383fff700 1 -- 172.21.15.178:0/929517272 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f138c00f4e0 con 0x7f139c0601e0 2023-11-15T04:57:18.943 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.941+0000 7f13a0b96700 1 -- 172.21.15.178:0/929517272 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f139c0fbcd0 con 0x7f139c0601e0 2023-11-15T04:57:18.943 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.942+0000 7f1383fff700 1 -- 172.21.15.178:0/929517272 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f138c00f300 con 0x7f139c0601e0 2023-11-15T04:57:18.944 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:57:18.944 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:57:18.945 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.944+0000 7f13a0b96700 1 -- 172.21.15.178:0/929517272 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f138402ff60 msgr2=0x7f1384032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:18.945 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.944+0000 7f13a0b96700 1 --2- 172.21.15.178:0/929517272 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f138402ff60 0x7f1384032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f13900096a0 tx=0x7f1390006b40).stop 2023-11-15T04:57:18.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.944+0000 7f13a0b96700 1 -- 172.21.15.178:0/929517272 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f139c0601e0 msgr2=0x7f139c0f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:18.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.944+0000 7f13a0b96700 1 --2- 172.21.15.178:0/929517272 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f139c0601e0 0x7f139c0f69a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f138c014000 tx=0x7f138c00b040).stop 2023-11-15T04:57:18.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.944+0000 7f13a0b96700 1 -- 172.21.15.178:0/929517272 shutdown_connections 2023-11-15T04:57:18.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.944+0000 7f13a0b96700 1 --2- 172.21.15.178:0/929517272 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f138402ff60 0x7f1384032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:18.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.944+0000 7f13a0b96700 1 --2- 172.21.15.178:0/929517272 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f139c0601e0 0x7f139c0f69a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:18.947 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.944+0000 7f13a0b96700 1 -- 172.21.15.178:0/929517272 >> 172.21.15.178:0/929517272 conn(0x7f139c0f09a0 msgr2=0x7f139c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:18.947 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.944+0000 7f13a0b96700 1 -- 172.21.15.178:0/929517272 shutdown_connections 2023-11-15T04:57:18.947 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:18.944+0000 7f13a0b96700 1 -- 172.21.15.178:0/929517272 wait complete. 2023-11-15T04:57:18.947 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:20.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:19 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:18.325521+0000 mgr.smithi163.qesyrx (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:20.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:19 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:18.943863+0000 mon.smithi163 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.178:0/929517272' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:20.343 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:20.343 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:57:21.532 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.530+0000 7ffafc6e9700 1 -- 172.21.15.178:0/734933341 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffaf40f5410 msgr2=0x7ffaf40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:21.533 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.530+0000 7ffafc6e9700 1 --2- 172.21.15.178:0/734933341 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffaf40f5410 0x7ffaf40f5830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7ffae4009fb0 tx=0x7ffae4009b30).stop 2023-11-15T04:57:21.533 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.530+0000 7ffafc6e9700 1 -- 172.21.15.178:0/734933341 shutdown_connections 2023-11-15T04:57:21.533 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.530+0000 7ffafc6e9700 1 --2- 172.21.15.178:0/734933341 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffaf40f5410 0x7ffaf40f5830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:21.533 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.530+0000 7ffafc6e9700 1 -- 172.21.15.178:0/734933341 >> 172.21.15.178:0/734933341 conn(0x7ffaf40f0b70 msgr2=0x7ffaf40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:21.534 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.530+0000 7ffafc6e9700 1 -- 172.21.15.178:0/734933341 shutdown_connections 2023-11-15T04:57:21.534 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.530+0000 7ffafc6e9700 1 -- 172.21.15.178:0/734933341 wait complete. 2023-11-15T04:57:21.534 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.531+0000 7ffafc6e9700 1 Processor -- start 2023-11-15T04:57:21.534 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.531+0000 7ffafc6e9700 1 -- start start 2023-11-15T04:57:21.534 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.531+0000 7ffafc6e9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffaf40f5410 0x7ffaf4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:21.535 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.531+0000 7ffafc6e9700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf4105b20 con 0x7ffaf40f5410 2023-11-15T04:57:21.535 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.531+0000 7ffafa485700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffaf40f5410 0x7ffaf4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:21.535 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.531+0000 7ffafa485700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffaf40f5410 0x7ffaf4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:39192/0 (socket says 172.21.15.178:39192) 2023-11-15T04:57:21.536 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.531+0000 7ffafa485700 1 -- 172.21.15.178:0/349120512 learned_addr learned my addr 172.21.15.178:0/349120512 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:57:21.536 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.531+0000 7ffafa485700 1 -- 172.21.15.178:0/349120512 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae4005040 con 0x7ffaf40f5410 2023-11-15T04:57:21.536 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.531+0000 7ffafa485700 1 --2- 172.21.15.178:0/349120512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffaf40f5410 0x7ffaf4105630 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7ffae4013510 tx=0x7ffae400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:21.537 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.532+0000 7ffaeaffd700 1 -- 172.21.15.178:0/349120512 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffae4009320 con 0x7ffaf40f5410 2023-11-15T04:57:21.537 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.532+0000 7ffafc6e9700 1 -- 172.21.15.178:0/349120512 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf4105d20 con 0x7ffaf40f5410 2023-11-15T04:57:21.538 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.532+0000 7ffaeaffd700 1 -- 172.21.15.178:0/349120512 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffae4013f00 con 0x7ffaf40f5410 2023-11-15T04:57:21.538 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.532+0000 7ffaeaffd700 1 -- 172.21.15.178:0/349120512 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffae4014d90 con 0x7ffaf40f5410 2023-11-15T04:57:21.538 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.532+0000 7ffafc6e9700 1 -- 172.21.15.178:0/349120512 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf41061b0 con 0x7ffaf40f5410 2023-11-15T04:57:21.538 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.532+0000 7ffaeaffd700 1 -- 172.21.15.178:0/349120512 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffae40095f0 con 0x7ffaf40f5410 2023-11-15T04:57:21.538 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.533+0000 7ffaeaffd700 1 --2- 172.21.15.178:0/349120512 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ffae002ff60 0x7ffae0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:21.539 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.533+0000 7ffafc6e9700 1 -- 172.21.15.178:0/349120512 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf4045ad0 con 0x7ffaf40f5410 2023-11-15T04:57:21.539 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.533+0000 7ffaeaffd700 1 -- 172.21.15.178:0/349120512 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffae4041000 con 0x7ffaf40f5410 2023-11-15T04:57:21.539 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.533+0000 7ffaf9c84700 1 --2- 172.21.15.178:0/349120512 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ffae002ff60 0x7ffae0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:21.539 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.533+0000 7ffaf9c84700 1 --2- 172.21.15.178:0/349120512 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ffae002ff60 0x7ffae0032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7ffaf00096a0 tx=0x7ffaf0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:21.540 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.538+0000 7ffaeaffd700 1 -- 172.21.15.178:0/349120512 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffae406d2b0 con 0x7ffaf40f5410 2023-11-15T04:57:21.745 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.743+0000 7ffafc6e9700 1 -- 172.21.15.178:0/349120512 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffaf40faf50 con 0x7ffaf40f5410 2023-11-15T04:57:21.745 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.744+0000 7ffaeaffd700 1 -- 172.21.15.178:0/349120512 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffae401d7f0 con 0x7ffaf40f5410 2023-11-15T04:57:21.746 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:57:21.746 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:57:21.747 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.746+0000 7ffafc6e9700 1 -- 172.21.15.178:0/349120512 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ffae002ff60 msgr2=0x7ffae0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:21.747 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.746+0000 7ffafc6e9700 1 --2- 172.21.15.178:0/349120512 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ffae002ff60 0x7ffae0032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7ffaf00096a0 tx=0x7ffaf0006b40).stop 2023-11-15T04:57:21.748 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.746+0000 7ffafc6e9700 1 -- 172.21.15.178:0/349120512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffaf40f5410 msgr2=0x7ffaf4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:21.748 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.746+0000 7ffafc6e9700 1 --2- 172.21.15.178:0/349120512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffaf40f5410 0x7ffaf4105630 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7ffae4013510 tx=0x7ffae400b040).stop 2023-11-15T04:57:21.748 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.746+0000 7ffafc6e9700 1 -- 172.21.15.178:0/349120512 shutdown_connections 2023-11-15T04:57:21.748 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.746+0000 7ffafc6e9700 1 --2- 172.21.15.178:0/349120512 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ffae002ff60 0x7ffae0032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:21.748 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.746+0000 7ffafc6e9700 1 --2- 172.21.15.178:0/349120512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffaf40f5410 0x7ffaf4105630 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:21.749 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.746+0000 7ffafc6e9700 1 -- 172.21.15.178:0/349120512 >> 172.21.15.178:0/349120512 conn(0x7ffaf40f0b70 msgr2=0x7ffaf40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:21.749 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.746+0000 7ffafc6e9700 1 -- 172.21.15.178:0/349120512 shutdown_connections 2023-11-15T04:57:21.749 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:21.746+0000 7ffafc6e9700 1 -- 172.21.15.178:0/349120512 wait complete. 2023-11-15T04:57:21.749 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:22.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:21 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:20.325828+0000 mgr.smithi163.qesyrx (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:22.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:21 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:21.745645+0000 mon.smithi163 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.178:0/349120512' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:23.145 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:23.146 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:57:24.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:23 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:22.326161+0000 mgr.smithi163.qesyrx (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:24.383 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.379+0000 7fb093dcf700 1 -- 172.21.15.178:0/2226425404 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb08c0f6510 msgr2=0x7fb08c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:24.383 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.379+0000 7fb093dcf700 1 --2- 172.21.15.178:0/2226425404 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb08c0f6510 0x7fb08c0f6930 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fb080009fb0 tx=0x7fb080009b30).stop 2023-11-15T04:57:24.384 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.380+0000 7fb093dcf700 1 -- 172.21.15.178:0/2226425404 shutdown_connections 2023-11-15T04:57:24.384 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.380+0000 7fb093dcf700 1 --2- 172.21.15.178:0/2226425404 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb08c0f6510 0x7fb08c0f6930 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:24.384 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.380+0000 7fb093dcf700 1 -- 172.21.15.178:0/2226425404 >> 172.21.15.178:0/2226425404 conn(0x7fb08c0f1c70 msgr2=0x7fb08c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:24.385 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.380+0000 7fb093dcf700 1 -- 172.21.15.178:0/2226425404 shutdown_connections 2023-11-15T04:57:24.385 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.380+0000 7fb093dcf700 1 -- 172.21.15.178:0/2226425404 wait complete. 2023-11-15T04:57:24.385 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.380+0000 7fb093dcf700 1 Processor -- start 2023-11-15T04:57:24.386 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.380+0000 7fb093dcf700 1 -- start start 2023-11-15T04:57:24.386 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.380+0000 7fb093dcf700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb08c0f6510 0x7fb08c1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:24.387 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.380+0000 7fb093dcf700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08c106bc0 con 0x7fb08c0f6510 2023-11-15T04:57:24.387 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.380+0000 7fb091b6b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb08c0f6510 0x7fb08c1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:24.387 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.381+0000 7fb091b6b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb08c0f6510 0x7fb08c1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:37950/0 (socket says 172.21.15.178:37950) 2023-11-15T04:57:24.388 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.381+0000 7fb091b6b700 1 -- 172.21.15.178:0/2526672992 learned_addr learned my addr 172.21.15.178:0/2526672992 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:57:24.388 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.381+0000 7fb091b6b700 1 -- 172.21.15.178:0/2526672992 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb080005040 con 0x7fb08c0f6510 2023-11-15T04:57:24.388 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.381+0000 7fb091b6b700 1 --2- 172.21.15.178:0/2526672992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb08c0f6510 0x7fb08c1066d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fb080014000 tx=0x7fb08000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:24.388 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.381+0000 7fb07e7fc700 1 -- 172.21.15.178:0/2526672992 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0800095f0 con 0x7fb08c0f6510 2023-11-15T04:57:24.389 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.381+0000 7fb093dcf700 1 -- 172.21.15.178:0/2526672992 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb08c106dc0 con 0x7fb08c0f6510 2023-11-15T04:57:24.389 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.381+0000 7fb07e7fc700 1 -- 172.21.15.178:0/2526672992 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb080007e70 con 0x7fb08c0f6510 2023-11-15T04:57:24.389 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.381+0000 7fb07e7fc700 1 -- 172.21.15.178:0/2526672992 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb080014ea0 con 0x7fb08c0f6510 2023-11-15T04:57:24.390 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.381+0000 7fb093dcf700 1 -- 172.21.15.178:0/2526672992 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb08c107250 con 0x7fb08c0f6510 2023-11-15T04:57:24.392 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.382+0000 7fb07e7fc700 1 -- 172.21.15.178:0/2526672992 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb080025070 con 0x7fb08c0f6510 2023-11-15T04:57:24.392 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.382+0000 7fb093dcf700 1 -- 172.21.15.178:0/2526672992 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb08c045ad0 con 0x7fb08c0f6510 2023-11-15T04:57:24.392 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.382+0000 7fb07e7fc700 1 --2- 172.21.15.178:0/2526672992 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb07802ff60 0x7fb078032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:24.392 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.382+0000 7fb07e7fc700 1 -- 172.21.15.178:0/2526672992 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb080040a30 con 0x7fb08c0f6510 2023-11-15T04:57:24.393 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.382+0000 7fb09136a700 1 --2- 172.21.15.178:0/2526672992 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb07802ff60 0x7fb078032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:24.393 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.383+0000 7fb09136a700 1 --2- 172.21.15.178:0/2526672992 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb07802ff60 0x7fb078032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fb0880096a0 tx=0x7fb088006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:24.393 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.387+0000 7fb07e7fc700 1 -- 172.21.15.178:0/2526672992 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb0800079c0 con 0x7fb08c0f6510 2023-11-15T04:57:24.593 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.592+0000 7fb093dcf700 1 -- 172.21.15.178:0/2526672992 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb08c108100 con 0x7fb08c0f6510 2023-11-15T04:57:24.594 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.592+0000 7fb07e7fc700 1 -- 172.21.15.178:0/2526672992 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb08000f300 con 0x7fb08c0f6510 2023-11-15T04:57:24.594 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:57:24.594 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:57:24.595 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.594+0000 7fb093dcf700 1 -- 172.21.15.178:0/2526672992 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb07802ff60 msgr2=0x7fb078032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:24.595 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.594+0000 7fb093dcf700 1 --2- 172.21.15.178:0/2526672992 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb07802ff60 0x7fb078032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fb0880096a0 tx=0x7fb088006b40).stop 2023-11-15T04:57:24.596 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.594+0000 7fb093dcf700 1 -- 172.21.15.178:0/2526672992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb08c0f6510 msgr2=0x7fb08c1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:24.596 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.594+0000 7fb093dcf700 1 --2- 172.21.15.178:0/2526672992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb08c0f6510 0x7fb08c1066d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fb080014000 tx=0x7fb08000b040).stop 2023-11-15T04:57:24.596 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.594+0000 7fb093dcf700 1 -- 172.21.15.178:0/2526672992 shutdown_connections 2023-11-15T04:57:24.596 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.594+0000 7fb093dcf700 1 --2- 172.21.15.178:0/2526672992 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb07802ff60 0x7fb078032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:24.596 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.594+0000 7fb093dcf700 1 --2- 172.21.15.178:0/2526672992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb08c0f6510 0x7fb08c1066d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:24.597 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.594+0000 7fb093dcf700 1 -- 172.21.15.178:0/2526672992 >> 172.21.15.178:0/2526672992 conn(0x7fb08c0f1c70 msgr2=0x7fb08c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:24.597 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.594+0000 7fb093dcf700 1 -- 172.21.15.178:0/2526672992 shutdown_connections 2023-11-15T04:57:24.597 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:24.594+0000 7fb093dcf700 1 -- 172.21.15.178:0/2526672992 wait complete. 2023-11-15T04:57:24.597 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:25.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:24 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:24.449344+0000 mon.smithi163 (mon.0) 150 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:57:25.206 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:24 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:24.451326+0000 mon.smithi163 (mon.0) 151 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/trash_purge_schedule"}]: dispatch 2023-11-15T04:57:25.206 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:24 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:24.594171+0000 mon.smithi163 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.178:0/2526672992' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:26.027 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:26.027 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:57:26.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:24.326542+0000 mgr.smithi163.qesyrx (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:27.247 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.242+0000 7fec20c59700 1 -- 172.21.15.178:0/302849014 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec1c0f51a0 msgr2=0x7fec1c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:27.248 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.242+0000 7fec20c59700 1 --2- 172.21.15.178:0/302849014 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec1c0f51a0 0x7fec1c0f55c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fec0c009fb0 tx=0x7fec0c009b30).stop 2023-11-15T04:57:27.248 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.243+0000 7fec20c59700 1 -- 172.21.15.178:0/302849014 shutdown_connections 2023-11-15T04:57:27.249 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.243+0000 7fec20c59700 1 --2- 172.21.15.178:0/302849014 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec1c0f51a0 0x7fec1c0f55c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:27.249 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.243+0000 7fec20c59700 1 -- 172.21.15.178:0/302849014 >> 172.21.15.178:0/302849014 conn(0x7fec1c0f09a0 msgr2=0x7fec1c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:27.249 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.243+0000 7fec20c59700 1 -- 172.21.15.178:0/302849014 shutdown_connections 2023-11-15T04:57:27.250 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.243+0000 7fec20c59700 1 -- 172.21.15.178:0/302849014 wait complete. 2023-11-15T04:57:27.250 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.243+0000 7fec20c59700 1 Processor -- start 2023-11-15T04:57:27.250 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.244+0000 7fec20c59700 1 -- start start 2023-11-15T04:57:27.251 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.244+0000 7fec20c59700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec1c0f51a0 0x7fec1c1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:27.251 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.244+0000 7fec20c59700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c1058a0 con 0x7fec1c0f51a0 2023-11-15T04:57:27.251 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.244+0000 7fec1b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec1c0f51a0 0x7fec1c1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:27.252 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.244+0000 7fec1b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec1c0f51a0 0x7fec1c1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:37970/0 (socket says 172.21.15.178:37970) 2023-11-15T04:57:27.252 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.244+0000 7fec1b7fe700 1 -- 172.21.15.178:0/19353396 learned_addr learned my addr 172.21.15.178:0/19353396 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:57:27.252 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.245+0000 7fec1b7fe700 1 -- 172.21.15.178:0/19353396 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0c005040 con 0x7fec1c0f51a0 2023-11-15T04:57:27.252 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.245+0000 7fec1b7fe700 1 --2- 172.21.15.178:0/19353396 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec1c0f51a0 0x7fec1c1053b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fec0c013510 tx=0x7fec0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:27.253 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.245+0000 7fec03fff700 1 -- 172.21.15.178:0/19353396 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec0c009320 con 0x7fec1c0f51a0 2023-11-15T04:57:27.253 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.245+0000 7fec03fff700 1 -- 172.21.15.178:0/19353396 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec0c013f00 con 0x7fec1c0f51a0 2023-11-15T04:57:27.253 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.245+0000 7fec20c59700 1 -- 172.21.15.178:0/19353396 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec1c105aa0 con 0x7fec1c0f51a0 2023-11-15T04:57:27.254 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.245+0000 7fec03fff700 1 -- 172.21.15.178:0/19353396 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec0c014dc0 con 0x7fec1c0f51a0 2023-11-15T04:57:27.254 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.245+0000 7fec20c59700 1 -- 172.21.15.178:0/19353396 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec1c105f30 con 0x7fec1c0f51a0 2023-11-15T04:57:27.254 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.246+0000 7fec03fff700 1 -- 172.21.15.178:0/19353396 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec0c014420 con 0x7fec1c0f51a0 2023-11-15T04:57:27.255 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.246+0000 7fec03fff700 1 --2- 172.21.15.178:0/19353396 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fec0402ff60 0x7fec04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:27.255 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.246+0000 7fec20c59700 1 -- 172.21.15.178:0/19353396 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec1c045ad0 con 0x7fec1c0f51a0 2023-11-15T04:57:27.255 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.246+0000 7fec03fff700 1 -- 172.21.15.178:0/19353396 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec0c041030 con 0x7fec1c0f51a0 2023-11-15T04:57:27.257 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.246+0000 7fec1affd700 1 --2- 172.21.15.178:0/19353396 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fec0402ff60 0x7fec04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:27.258 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.247+0000 7fec1affd700 1 --2- 172.21.15.178:0/19353396 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fec0402ff60 0x7fec04032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fec100096a0 tx=0x7fec10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:27.258 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.251+0000 7fec03fff700 1 -- 172.21.15.178:0/19353396 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec0c0095f0 con 0x7fec1c0f51a0 2023-11-15T04:57:27.458 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.456+0000 7fec20c59700 1 -- 172.21.15.178:0/19353396 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec1c106de0 con 0x7fec1c0f51a0 2023-11-15T04:57:27.458 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.457+0000 7fec03fff700 1 -- 172.21.15.178:0/19353396 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fec0c01cb10 con 0x7fec1c0f51a0 2023-11-15T04:57:27.459 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:57:27.459 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:57:27.462 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.458+0000 7fec20c59700 1 -- 172.21.15.178:0/19353396 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fec0402ff60 msgr2=0x7fec04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:27.462 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.458+0000 7fec20c59700 1 --2- 172.21.15.178:0/19353396 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fec0402ff60 0x7fec04032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fec100096a0 tx=0x7fec10006b40).stop 2023-11-15T04:57:27.463 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.458+0000 7fec20c59700 1 -- 172.21.15.178:0/19353396 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec1c0f51a0 msgr2=0x7fec1c1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:27.463 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.458+0000 7fec20c59700 1 --2- 172.21.15.178:0/19353396 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec1c0f51a0 0x7fec1c1053b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fec0c013510 tx=0x7fec0c00b040).stop 2023-11-15T04:57:27.463 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.458+0000 7fec20c59700 1 -- 172.21.15.178:0/19353396 shutdown_connections 2023-11-15T04:57:27.464 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.458+0000 7fec20c59700 1 --2- 172.21.15.178:0/19353396 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fec0402ff60 0x7fec04032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:27.465 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.458+0000 7fec20c59700 1 --2- 172.21.15.178:0/19353396 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec1c0f51a0 0x7fec1c1053b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:27.465 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.458+0000 7fec20c59700 1 -- 172.21.15.178:0/19353396 >> 172.21.15.178:0/19353396 conn(0x7fec1c0f09a0 msgr2=0x7fec1c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:27.465 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.459+0000 7fec20c59700 1 -- 172.21.15.178:0/19353396 shutdown_connections 2023-11-15T04:57:27.465 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:27.459+0000 7fec20c59700 1 -- 172.21.15.178:0/19353396 wait complete. 2023-11-15T04:57:27.466 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:28.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:27 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:26.326835+0000 mgr.smithi163.qesyrx (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:28.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:27 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:27.458709+0000 mon.smithi163 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.178:0/19353396' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:28.884 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:28.885 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:57:30.055 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.053+0000 7f76c1da6700 1 -- 172.21.15.178:0/1169356271 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76bc0f5410 msgr2=0x7f76bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:30.055 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.053+0000 7f76c1da6700 1 --2- 172.21.15.178:0/1169356271 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76bc0f5410 0x7f76bc0f5830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f76ac009fb0 tx=0x7f76ac009b30).stop 2023-11-15T04:57:30.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.053+0000 7f76c1da6700 1 -- 172.21.15.178:0/1169356271 shutdown_connections 2023-11-15T04:57:30.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.053+0000 7f76c1da6700 1 --2- 172.21.15.178:0/1169356271 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76bc0f5410 0x7f76bc0f5830 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:30.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.053+0000 7f76c1da6700 1 -- 172.21.15.178:0/1169356271 >> 172.21.15.178:0/1169356271 conn(0x7f76bc0f0b70 msgr2=0x7f76bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:30.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.053+0000 7f76c1da6700 1 -- 172.21.15.178:0/1169356271 shutdown_connections 2023-11-15T04:57:30.057 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.053+0000 7f76c1da6700 1 -- 172.21.15.178:0/1169356271 wait complete. 2023-11-15T04:57:30.057 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.053+0000 7f76c1da6700 1 Processor -- start 2023-11-15T04:57:30.057 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.053+0000 7f76c1da6700 1 -- start start 2023-11-15T04:57:30.057 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.054+0000 7f76c1da6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76bc0f5410 0x7f76bc109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:30.058 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.054+0000 7f76c1da6700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc060240 con 0x7f76bc0f5410 2023-11-15T04:57:30.059 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.054+0000 7f76bb7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76bc0f5410 0x7f76bc109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:30.060 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.054+0000 7f76bb7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76bc0f5410 0x7f76bc109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:37986/0 (socket says 172.21.15.178:37986) 2023-11-15T04:57:30.060 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.054+0000 7f76bb7fe700 1 -- 172.21.15.178:0/807680759 learned_addr learned my addr 172.21.15.178:0/807680759 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:57:30.060 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.054+0000 7f76bb7fe700 1 -- 172.21.15.178:0/807680759 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ac005040 con 0x7f76bc0f5410 2023-11-15T04:57:30.060 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.054+0000 7f76bb7fe700 1 --2- 172.21.15.178:0/807680759 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76bc0f5410 0x7f76bc109400 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f76ac013510 tx=0x7f76ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:30.061 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.055+0000 7f76a3fff700 1 -- 172.21.15.178:0/807680759 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76ac009320 con 0x7f76bc0f5410 2023-11-15T04:57:30.061 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.055+0000 7f76a3fff700 1 -- 172.21.15.178:0/807680759 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f76ac013f00 con 0x7f76bc0f5410 2023-11-15T04:57:30.061 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.055+0000 7f76c1da6700 1 -- 172.21.15.178:0/807680759 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76bc060440 con 0x7f76bc0f5410 2023-11-15T04:57:30.061 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.055+0000 7f76a3fff700 1 -- 172.21.15.178:0/807680759 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76ac014d90 con 0x7f76bc0f5410 2023-11-15T04:57:30.062 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.055+0000 7f76c1da6700 1 -- 172.21.15.178:0/807680759 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76bc109ab0 con 0x7f76bc0f5410 2023-11-15T04:57:30.062 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.055+0000 7f76a3fff700 1 -- 172.21.15.178:0/807680759 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f76ac014420 con 0x7f76bc0f5410 2023-11-15T04:57:30.062 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.056+0000 7f76a3fff700 1 --2- 172.21.15.178:0/807680759 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f76a402ff60 0x7f76a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:30.062 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.056+0000 7f76c1da6700 1 -- 172.21.15.178:0/807680759 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76bc045ad0 con 0x7f76bc0f5410 2023-11-15T04:57:30.062 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.056+0000 7f76a3fff700 1 -- 172.21.15.178:0/807680759 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f76ac0410c0 con 0x7f76bc0f5410 2023-11-15T04:57:30.063 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.056+0000 7f76baffd700 1 --2- 172.21.15.178:0/807680759 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f76a402ff60 0x7f76a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:30.063 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.056+0000 7f76baffd700 1 --2- 172.21.15.178:0/807680759 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f76a402ff60 0x7f76a4032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f76b00096a0 tx=0x7f76b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:30.064 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.061+0000 7f76a3fff700 1 -- 172.21.15.178:0/807680759 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f76ac06d2b0 con 0x7f76bc0f5410 2023-11-15T04:57:30.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:29 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:28.327194+0000 mgr.smithi163.qesyrx (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:30.269 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.267+0000 7f76c1da6700 1 -- 172.21.15.178:0/807680759 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76bc0faf50 con 0x7f76bc0f5410 2023-11-15T04:57:30.270 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.268+0000 7f76a3fff700 1 -- 172.21.15.178:0/807680759 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f76ac01dd50 con 0x7f76bc0f5410 2023-11-15T04:57:30.270 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:57:30.271 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:57:30.272 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.270+0000 7f76c1da6700 1 -- 172.21.15.178:0/807680759 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f76a402ff60 msgr2=0x7f76a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:30.272 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.270+0000 7f76c1da6700 1 --2- 172.21.15.178:0/807680759 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f76a402ff60 0x7f76a4032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f76b00096a0 tx=0x7f76b0006b40).stop 2023-11-15T04:57:30.272 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.270+0000 7f76c1da6700 1 -- 172.21.15.178:0/807680759 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76bc0f5410 msgr2=0x7f76bc109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:30.272 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.270+0000 7f76c1da6700 1 --2- 172.21.15.178:0/807680759 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76bc0f5410 0x7f76bc109400 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f76ac013510 tx=0x7f76ac00b040).stop 2023-11-15T04:57:30.273 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.270+0000 7f76c1da6700 1 -- 172.21.15.178:0/807680759 shutdown_connections 2023-11-15T04:57:30.273 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.270+0000 7f76c1da6700 1 --2- 172.21.15.178:0/807680759 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f76a402ff60 0x7f76a4032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:30.273 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.270+0000 7f76c1da6700 1 --2- 172.21.15.178:0/807680759 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76bc0f5410 0x7f76bc109400 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:30.273 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.270+0000 7f76c1da6700 1 -- 172.21.15.178:0/807680759 >> 172.21.15.178:0/807680759 conn(0x7f76bc0f0b70 msgr2=0x7f76bc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:30.273 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.270+0000 7f76c1da6700 1 -- 172.21.15.178:0/807680759 shutdown_connections 2023-11-15T04:57:30.274 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:30.270+0000 7f76c1da6700 1 -- 172.21.15.178:0/807680759 wait complete. 2023-11-15T04:57:30.274 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:31.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:30 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:30.269823+0000 mon.smithi163 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.178:0/807680759' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:31.678 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:31.679 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:57:32.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:31 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:30.327524+0000 mgr.smithi163.qesyrx (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:32.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.889+0000 7f3eb5437700 1 -- 172.21.15.178:0/1063822786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3eb00f51a0 msgr2=0x7f3eb00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:32.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.889+0000 7f3eb5437700 1 --2- 172.21.15.178:0/1063822786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3eb00f51a0 0x7f3eb00f55c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f3ea0009fb0 tx=0x7f3ea0009b30).stop 2023-11-15T04:57:32.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.890+0000 7f3eb5437700 1 -- 172.21.15.178:0/1063822786 shutdown_connections 2023-11-15T04:57:32.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.890+0000 7f3eb5437700 1 --2- 172.21.15.178:0/1063822786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3eb00f51a0 0x7f3eb00f55c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:32.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.890+0000 7f3eb5437700 1 -- 172.21.15.178:0/1063822786 >> 172.21.15.178:0/1063822786 conn(0x7f3eb00f09a0 msgr2=0x7f3eb00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:32.893 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.890+0000 7f3eb5437700 1 -- 172.21.15.178:0/1063822786 shutdown_connections 2023-11-15T04:57:32.893 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.890+0000 7f3eb5437700 1 -- 172.21.15.178:0/1063822786 wait complete. 2023-11-15T04:57:32.893 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.890+0000 7f3eb5437700 1 Processor -- start 2023-11-15T04:57:32.895 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.890+0000 7f3eb5437700 1 -- start start 2023-11-15T04:57:32.895 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.891+0000 7f3eb5437700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3eb00f51a0 0x7f3eb01042a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:32.895 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.891+0000 7f3eb5437700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb0104790 con 0x7f3eb00f51a0 2023-11-15T04:57:32.896 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.891+0000 7f3eaffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3eb00f51a0 0x7f3eb01042a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:32.896 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.891+0000 7f3eaffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3eb00f51a0 0x7f3eb01042a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:38010/0 (socket says 172.21.15.178:38010) 2023-11-15T04:57:32.896 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.891+0000 7f3eaffff700 1 -- 172.21.15.178:0/1422255535 learned_addr learned my addr 172.21.15.178:0/1422255535 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:57:32.896 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.892+0000 7f3eaffff700 1 -- 172.21.15.178:0/1422255535 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea0005040 con 0x7f3eb00f51a0 2023-11-15T04:57:32.896 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.892+0000 7f3eaffff700 1 --2- 172.21.15.178:0/1422255535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3eb00f51a0 0x7f3eb01042a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f3ea0013510 tx=0x7f3ea000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:32.897 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.892+0000 7f3eacff9700 1 -- 172.21.15.178:0/1422255535 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ea0009320 con 0x7f3eb00f51a0 2023-11-15T04:57:32.897 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.892+0000 7f3eb5437700 1 -- 172.21.15.178:0/1422255535 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb0104990 con 0x7f3eb00f51a0 2023-11-15T04:57:32.897 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.892+0000 7f3eacff9700 1 -- 172.21.15.178:0/1422255535 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ea0013f00 con 0x7f3eb00f51a0 2023-11-15T04:57:32.897 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.892+0000 7f3eacff9700 1 -- 172.21.15.178:0/1422255535 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ea0014dc0 con 0x7f3eb00f51a0 2023-11-15T04:57:32.897 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.892+0000 7f3eb5437700 1 -- 172.21.15.178:0/1422255535 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb0104e20 con 0x7f3eb00f51a0 2023-11-15T04:57:32.898 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.893+0000 7f3eacff9700 1 -- 172.21.15.178:0/1422255535 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ea00263a0 con 0x7f3eb00f51a0 2023-11-15T04:57:32.898 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.893+0000 7f3eacff9700 1 --2- 172.21.15.178:0/1422255535 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3e980321b0 0x7f3e98034670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:32.898 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.893+0000 7f3eacff9700 1 -- 172.21.15.178:0/1422255535 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ea0040ff0 con 0x7f3eb00f51a0 2023-11-15T04:57:32.899 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.893+0000 7f3eb5437700 1 -- 172.21.15.178:0/1422255535 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb0045ad0 con 0x7f3eb00f51a0 2023-11-15T04:57:32.899 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.893+0000 7f3eaf7fe700 1 --2- 172.21.15.178:0/1422255535 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3e980321b0 0x7f3e98034670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:32.899 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.894+0000 7f3eaf7fe700 1 --2- 172.21.15.178:0/1422255535 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3e980321b0 0x7f3e98034670 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f3ea40096a0 tx=0x7f3ea4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:32.899 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:32.898+0000 7f3eacff9700 1 -- 172.21.15.178:0/1422255535 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ea0012920 con 0x7f3eb00f51a0 2023-11-15T04:57:33.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:33.102+0000 7f3eb5437700 1 -- 172.21.15.178:0/1422255535 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3eb0105b40 con 0x7f3eb00f51a0 2023-11-15T04:57:33.104 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:33.102+0000 7f3eacff9700 1 -- 172.21.15.178:0/1422255535 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3ea002b020 con 0x7f3eb00f51a0 2023-11-15T04:57:33.104 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:57:33.104 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:57:33.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:33.104+0000 7f3eb5437700 1 -- 172.21.15.178:0/1422255535 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3e980321b0 msgr2=0x7f3e98034670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:33.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:33.104+0000 7f3eb5437700 1 --2- 172.21.15.178:0/1422255535 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3e980321b0 0x7f3e98034670 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f3ea40096a0 tx=0x7f3ea4006b40).stop 2023-11-15T04:57:33.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:33.104+0000 7f3eb5437700 1 -- 172.21.15.178:0/1422255535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3eb00f51a0 msgr2=0x7f3eb01042a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:33.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:33.104+0000 7f3eb5437700 1 --2- 172.21.15.178:0/1422255535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3eb00f51a0 0x7f3eb01042a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f3ea0013510 tx=0x7f3ea000b040).stop 2023-11-15T04:57:33.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:33.104+0000 7f3eb5437700 1 -- 172.21.15.178:0/1422255535 shutdown_connections 2023-11-15T04:57:33.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:33.104+0000 7f3eb5437700 1 --2- 172.21.15.178:0/1422255535 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3e980321b0 0x7f3e98034670 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:33.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:33.104+0000 7f3eb5437700 1 --2- 172.21.15.178:0/1422255535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3eb00f51a0 0x7f3eb01042a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:33.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:33.104+0000 7f3eb5437700 1 -- 172.21.15.178:0/1422255535 >> 172.21.15.178:0/1422255535 conn(0x7f3eb00f09a0 msgr2=0x7f3eb00f1400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:33.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:33.104+0000 7f3eb5437700 1 -- 172.21.15.178:0/1422255535 shutdown_connections 2023-11-15T04:57:33.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:33.104+0000 7f3eb5437700 1 -- 172.21.15.178:0/1422255535 wait complete. 2023-11-15T04:57:33.108 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:34.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:33 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:32.327867+0000 mgr.smithi163.qesyrx (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:34.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:33 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:33.104030+0000 mon.smithi163 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.178:0/1422255535' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:34.479 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:34.479 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:57:35.664 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.660+0000 7f214ce4a700 1 -- 172.21.15.178:0/3060780041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21480f6510 msgr2=0x7f21480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:35.665 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.660+0000 7f214ce4a700 1 --2- 172.21.15.178:0/3060780041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21480f6510 0x7f21480f6930 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f2130009fb0 tx=0x7f2130009b30).stop 2023-11-15T04:57:35.665 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.660+0000 7f214ce4a700 1 -- 172.21.15.178:0/3060780041 shutdown_connections 2023-11-15T04:57:35.665 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.660+0000 7f214ce4a700 1 --2- 172.21.15.178:0/3060780041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21480f6510 0x7f21480f6930 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:35.665 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.660+0000 7f214ce4a700 1 -- 172.21.15.178:0/3060780041 >> 172.21.15.178:0/3060780041 conn(0x7f21480f1c70 msgr2=0x7f21480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:35.665 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.660+0000 7f214ce4a700 1 -- 172.21.15.178:0/3060780041 shutdown_connections 2023-11-15T04:57:35.666 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.660+0000 7f214ce4a700 1 -- 172.21.15.178:0/3060780041 wait complete. 2023-11-15T04:57:35.666 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.661+0000 7f214ce4a700 1 Processor -- start 2023-11-15T04:57:35.666 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.661+0000 7f214ce4a700 1 -- start start 2023-11-15T04:57:35.666 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.661+0000 7f214ce4a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21480f6510 0x7f214806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:35.666 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.661+0000 7f214ce4a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2148069e20 con 0x7f21480f6510 2023-11-15T04:57:35.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.661+0000 7f214659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21480f6510 0x7f214806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:35.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.661+0000 7f214659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21480f6510 0x7f214806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:58518/0 (socket says 172.21.15.178:58518) 2023-11-15T04:57:35.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.661+0000 7f214659c700 1 -- 172.21.15.178:0/1677433934 learned_addr learned my addr 172.21.15.178:0/1677433934 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:57:35.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.661+0000 7f214659c700 1 -- 172.21.15.178:0/1677433934 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2130005040 con 0x7f21480f6510 2023-11-15T04:57:35.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.661+0000 7f214659c700 1 --2- 172.21.15.178:0/1677433934 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21480f6510 0x7f214806b9a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f2130013510 tx=0x7f213000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:35.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.662+0000 7f213effd700 1 -- 172.21.15.178:0/1677433934 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2130009320 con 0x7f21480f6510 2023-11-15T04:57:35.668 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.662+0000 7f214ce4a700 1 -- 172.21.15.178:0/1677433934 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2148069fc0 con 0x7f21480f6510 2023-11-15T04:57:35.668 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.662+0000 7f213effd700 1 -- 172.21.15.178:0/1677433934 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2130013f00 con 0x7f21480f6510 2023-11-15T04:57:35.668 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.662+0000 7f213effd700 1 -- 172.21.15.178:0/1677433934 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2130014d90 con 0x7f21480f6510 2023-11-15T04:57:35.668 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.662+0000 7f214ce4a700 1 -- 172.21.15.178:0/1677433934 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f214806a450 con 0x7f21480f6510 2023-11-15T04:57:35.668 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.662+0000 7f213effd700 1 -- 172.21.15.178:0/1677433934 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2130014420 con 0x7f21480f6510 2023-11-15T04:57:35.669 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.663+0000 7f214ce4a700 1 -- 172.21.15.178:0/1677433934 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2148059100 con 0x7f21480f6510 2023-11-15T04:57:35.669 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.663+0000 7f213effd700 1 --2- 172.21.15.178:0/1677433934 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f213402ff60 0x7f2134032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:35.669 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.663+0000 7f213effd700 1 -- 172.21.15.178:0/1677433934 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2130040ff0 con 0x7f21480f6510 2023-11-15T04:57:35.669 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.663+0000 7f2145d9b700 1 --2- 172.21.15.178:0/1677433934 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f213402ff60 0x7f2134032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:35.669 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.663+0000 7f2145d9b700 1 --2- 172.21.15.178:0/1677433934 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f213402ff60 0x7f2134032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f21380096a0 tx=0x7f2138006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:35.670 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.668+0000 7f213effd700 1 -- 172.21.15.178:0/1677433934 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2130026bf0 con 0x7f21480f6510 2023-11-15T04:57:35.876 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.875+0000 7f214ce4a700 1 -- 172.21.15.178:0/1677433934 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21480fbf60 con 0x7f21480f6510 2023-11-15T04:57:35.877 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.875+0000 7f213effd700 1 -- 172.21.15.178:0/1677433934 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f213001db70 con 0x7f21480f6510 2023-11-15T04:57:35.878 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:57:35.878 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:57:35.879 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.877+0000 7f214ce4a700 1 -- 172.21.15.178:0/1677433934 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f213402ff60 msgr2=0x7f2134032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:35.879 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.877+0000 7f214ce4a700 1 --2- 172.21.15.178:0/1677433934 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f213402ff60 0x7f2134032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f21380096a0 tx=0x7f2138006b40).stop 2023-11-15T04:57:35.879 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.877+0000 7f214ce4a700 1 -- 172.21.15.178:0/1677433934 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21480f6510 msgr2=0x7f214806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:35.880 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.877+0000 7f214ce4a700 1 --2- 172.21.15.178:0/1677433934 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21480f6510 0x7f214806b9a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f2130013510 tx=0x7f213000b040).stop 2023-11-15T04:57:35.880 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.877+0000 7f214ce4a700 1 -- 172.21.15.178:0/1677433934 shutdown_connections 2023-11-15T04:57:35.880 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.877+0000 7f214ce4a700 1 --2- 172.21.15.178:0/1677433934 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f213402ff60 0x7f2134032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:35.880 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.877+0000 7f214ce4a700 1 --2- 172.21.15.178:0/1677433934 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21480f6510 0x7f214806b9a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:35.880 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.877+0000 7f214ce4a700 1 -- 172.21.15.178:0/1677433934 >> 172.21.15.178:0/1677433934 conn(0x7f21480f1c70 msgr2=0x7f21480f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:35.881 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.877+0000 7f214ce4a700 1 -- 172.21.15.178:0/1677433934 shutdown_connections 2023-11-15T04:57:35.881 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:35.877+0000 7f214ce4a700 1 -- 172.21.15.178:0/1677433934 wait complete. 2023-11-15T04:57:35.881 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:36.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:34.328217+0000 mgr.smithi163.qesyrx (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:37.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:35.876993+0000 mon.smithi163 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.178:0/1677433934' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:37.258 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:37.259 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:57:38.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:37 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:36.328567+0000 mgr.smithi163.qesyrx (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:38.405 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.403+0000 7f2177ba9700 1 -- 172.21.15.178:0/4100816930 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21700f5410 msgr2=0x7f21700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:38.406 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.403+0000 7f2177ba9700 1 --2- 172.21.15.178:0/4100816930 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21700f5410 0x7f21700f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f2164009fb0 tx=0x7f2164009b30).stop 2023-11-15T04:57:38.406 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.403+0000 7f2177ba9700 1 -- 172.21.15.178:0/4100816930 shutdown_connections 2023-11-15T04:57:38.406 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.403+0000 7f2177ba9700 1 --2- 172.21.15.178:0/4100816930 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21700f5410 0x7f21700f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:38.406 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.403+0000 7f2177ba9700 1 -- 172.21.15.178:0/4100816930 >> 172.21.15.178:0/4100816930 conn(0x7f21700f0b70 msgr2=0x7f21700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:38.407 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.403+0000 7f2177ba9700 1 -- 172.21.15.178:0/4100816930 shutdown_connections 2023-11-15T04:57:38.407 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.403+0000 7f2177ba9700 1 -- 172.21.15.178:0/4100816930 wait complete. 2023-11-15T04:57:38.407 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.404+0000 7f2177ba9700 1 Processor -- start 2023-11-15T04:57:38.407 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.404+0000 7f2177ba9700 1 -- start start 2023-11-15T04:57:38.407 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.404+0000 7f2177ba9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21700f5410 0x7f2170105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:38.408 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.404+0000 7f2177ba9700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2170105b20 con 0x7f21700f5410 2023-11-15T04:57:38.408 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.404+0000 7f2175945700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21700f5410 0x7f2170105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:38.408 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.404+0000 7f2175945700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21700f5410 0x7f2170105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:58524/0 (socket says 172.21.15.178:58524) 2023-11-15T04:57:38.408 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.404+0000 7f2175945700 1 -- 172.21.15.178:0/2885871163 learned_addr learned my addr 172.21.15.178:0/2885871163 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:57:38.410 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.405+0000 7f2175945700 1 -- 172.21.15.178:0/2885871163 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2164005040 con 0x7f21700f5410 2023-11-15T04:57:38.410 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.405+0000 7f2175945700 1 --2- 172.21.15.178:0/2885871163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21700f5410 0x7f2170105630 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f2164014000 tx=0x7f216400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:38.410 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.405+0000 7f21627fc700 1 -- 172.21.15.178:0/2885871163 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21640095f0 con 0x7f21700f5410 2023-11-15T04:57:38.410 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.405+0000 7f21627fc700 1 -- 172.21.15.178:0/2885871163 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2164007e70 con 0x7f21700f5410 2023-11-15T04:57:38.411 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.405+0000 7f2177ba9700 1 -- 172.21.15.178:0/2885871163 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2170105d20 con 0x7f21700f5410 2023-11-15T04:57:38.411 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.405+0000 7f21627fc700 1 -- 172.21.15.178:0/2885871163 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2164014ea0 con 0x7f21700f5410 2023-11-15T04:57:38.411 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.405+0000 7f2177ba9700 1 -- 172.21.15.178:0/2885871163 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21701061b0 con 0x7f21700f5410 2023-11-15T04:57:38.411 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.406+0000 7f21627fc700 1 -- 172.21.15.178:0/2885871163 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2164025070 con 0x7f21700f5410 2023-11-15T04:57:38.411 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.406+0000 7f2177ba9700 1 -- 172.21.15.178:0/2885871163 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2170045ad0 con 0x7f21700f5410 2023-11-15T04:57:38.411 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.406+0000 7f21627fc700 1 --2- 172.21.15.178:0/2885871163 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f215c02ff60 0x7f215c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:38.412 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.406+0000 7f21627fc700 1 -- 172.21.15.178:0/2885871163 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2164040930 con 0x7f21700f5410 2023-11-15T04:57:38.412 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.406+0000 7f2175144700 1 --2- 172.21.15.178:0/2885871163 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f215c02ff60 0x7f215c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:38.412 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.407+0000 7f2175144700 1 --2- 172.21.15.178:0/2885871163 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f215c02ff60 0x7f215c032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f216c0096a0 tx=0x7f216c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:38.413 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.411+0000 7f21627fc700 1 -- 172.21.15.178:0/2885871163 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f216400f4e0 con 0x7f21700f5410 2023-11-15T04:57:38.618 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.617+0000 7f2177ba9700 1 -- 172.21.15.178:0/2885871163 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21700fae60 con 0x7f21700f5410 2023-11-15T04:57:38.619 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.618+0000 7f21627fc700 1 -- 172.21.15.178:0/2885871163 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f216400f300 con 0x7f21700f5410 2023-11-15T04:57:38.619 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:57:38.620 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:57:38.621 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.620+0000 7f2177ba9700 1 -- 172.21.15.178:0/2885871163 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f215c02ff60 msgr2=0x7f215c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:38.621 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.620+0000 7f2177ba9700 1 --2- 172.21.15.178:0/2885871163 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f215c02ff60 0x7f215c032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f216c0096a0 tx=0x7f216c006b40).stop 2023-11-15T04:57:38.622 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.620+0000 7f2177ba9700 1 -- 172.21.15.178:0/2885871163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21700f5410 msgr2=0x7f2170105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:38.622 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.620+0000 7f2177ba9700 1 --2- 172.21.15.178:0/2885871163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21700f5410 0x7f2170105630 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f2164014000 tx=0x7f216400b040).stop 2023-11-15T04:57:38.622 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.620+0000 7f2177ba9700 1 -- 172.21.15.178:0/2885871163 shutdown_connections 2023-11-15T04:57:38.622 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.620+0000 7f2177ba9700 1 --2- 172.21.15.178:0/2885871163 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f215c02ff60 0x7f215c032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:38.622 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.620+0000 7f2177ba9700 1 --2- 172.21.15.178:0/2885871163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21700f5410 0x7f2170105630 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:38.623 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.620+0000 7f2177ba9700 1 -- 172.21.15.178:0/2885871163 >> 172.21.15.178:0/2885871163 conn(0x7f21700f0b70 msgr2=0x7f21700f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:38.623 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.620+0000 7f2177ba9700 1 -- 172.21.15.178:0/2885871163 shutdown_connections 2023-11-15T04:57:38.623 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:38.620+0000 7f2177ba9700 1 -- 172.21.15.178:0/2885871163 wait complete. 2023-11-15T04:57:38.623 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:39.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:38 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:38.619492+0000 mon.smithi163 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.178:0/2885871163' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:40.011 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:40.011 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:57:40.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:39 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:38.328858+0000 mgr.smithi163.qesyrx (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:41.275 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.273+0000 7f633f2cd700 1 -- 172.21.15.178:0/1709677395 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63380f6510 msgr2=0x7f63380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:41.275 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.273+0000 7f633f2cd700 1 --2- 172.21.15.178:0/1709677395 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63380f6510 0x7f63380f6930 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f6328009fb0 tx=0x7f6328009b30).stop 2023-11-15T04:57:41.275 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.273+0000 7f633f2cd700 1 -- 172.21.15.178:0/1709677395 shutdown_connections 2023-11-15T04:57:41.276 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.273+0000 7f633f2cd700 1 --2- 172.21.15.178:0/1709677395 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63380f6510 0x7f63380f6930 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:41.276 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.273+0000 7f633f2cd700 1 -- 172.21.15.178:0/1709677395 >> 172.21.15.178:0/1709677395 conn(0x7f63380f1c70 msgr2=0x7f63380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:41.276 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.273+0000 7f633f2cd700 1 -- 172.21.15.178:0/1709677395 shutdown_connections 2023-11-15T04:57:41.276 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.273+0000 7f633f2cd700 1 -- 172.21.15.178:0/1709677395 wait complete. 2023-11-15T04:57:41.276 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.273+0000 7f633f2cd700 1 Processor -- start 2023-11-15T04:57:41.277 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.274+0000 7f633f2cd700 1 -- start start 2023-11-15T04:57:41.277 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.274+0000 7f633f2cd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63380f6510 0x7f633806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:41.277 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.274+0000 7f633f2cd700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6338069e20 con 0x7f63380f6510 2023-11-15T04:57:41.279 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.274+0000 7f633d069700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63380f6510 0x7f633806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:41.279 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.274+0000 7f633d069700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63380f6510 0x7f633806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:58532/0 (socket says 172.21.15.178:58532) 2023-11-15T04:57:41.279 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.274+0000 7f633d069700 1 -- 172.21.15.178:0/3359652821 learned_addr learned my addr 172.21.15.178:0/3359652821 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:57:41.279 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.274+0000 7f633d069700 1 -- 172.21.15.178:0/3359652821 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6328005040 con 0x7f63380f6510 2023-11-15T04:57:41.279 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.274+0000 7f633d069700 1 --2- 172.21.15.178:0/3359652821 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63380f6510 0x7f633806b9a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f6328013510 tx=0x7f632800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:41.280 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.275+0000 7f632dffb700 1 -- 172.21.15.178:0/3359652821 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6328009320 con 0x7f63380f6510 2023-11-15T04:57:41.280 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.275+0000 7f633f2cd700 1 -- 172.21.15.178:0/3359652821 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6338069fc0 con 0x7f63380f6510 2023-11-15T04:57:41.280 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.275+0000 7f633f2cd700 1 -- 172.21.15.178:0/3359652821 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f633806a450 con 0x7f63380f6510 2023-11-15T04:57:41.280 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.275+0000 7f632dffb700 1 -- 172.21.15.178:0/3359652821 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6328013f00 con 0x7f63380f6510 2023-11-15T04:57:41.281 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.275+0000 7f632dffb700 1 -- 172.21.15.178:0/3359652821 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6328014d60 con 0x7f63380f6510 2023-11-15T04:57:41.281 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.276+0000 7f632dffb700 1 -- 172.21.15.178:0/3359652821 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6328014420 con 0x7f63380f6510 2023-11-15T04:57:41.281 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.276+0000 7f633f2cd700 1 -- 172.21.15.178:0/3359652821 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6338059100 con 0x7f63380f6510 2023-11-15T04:57:41.281 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.276+0000 7f632dffb700 1 --2- 172.21.15.178:0/3359652821 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f632402ffb0 0x7f6324032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:41.281 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.276+0000 7f632dffb700 1 -- 172.21.15.178:0/3359652821 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6328041360 con 0x7f63380f6510 2023-11-15T04:57:41.282 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.276+0000 7f633c868700 1 --2- 172.21.15.178:0/3359652821 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f632402ffb0 0x7f6324032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:41.282 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.277+0000 7f633c868700 1 --2- 172.21.15.178:0/3359652821 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f632402ffb0 0x7f6324032470 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f63340096a0 tx=0x7f6334006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:41.282 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.282+0000 7f632dffb700 1 -- 172.21.15.178:0/3359652821 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f632801c320 con 0x7f63380f6510 2023-11-15T04:57:41.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.484+0000 7f633f2cd700 1 -- 172.21.15.178:0/3359652821 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63380fbf60 con 0x7f63380f6510 2023-11-15T04:57:41.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.486+0000 7f632dffb700 1 -- 172.21.15.178:0/3359652821 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6328012b20 con 0x7f63380f6510 2023-11-15T04:57:41.487 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:57:41.487 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:57:41.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.487+0000 7f633f2cd700 1 -- 172.21.15.178:0/3359652821 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f632402ffb0 msgr2=0x7f6324032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:41.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.487+0000 7f633f2cd700 1 --2- 172.21.15.178:0/3359652821 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f632402ffb0 0x7f6324032470 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f63340096a0 tx=0x7f6334006b40).stop 2023-11-15T04:57:41.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.487+0000 7f633f2cd700 1 -- 172.21.15.178:0/3359652821 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63380f6510 msgr2=0x7f633806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:41.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.487+0000 7f633f2cd700 1 --2- 172.21.15.178:0/3359652821 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63380f6510 0x7f633806b9a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f6328013510 tx=0x7f632800b040).stop 2023-11-15T04:57:41.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.487+0000 7f633f2cd700 1 -- 172.21.15.178:0/3359652821 shutdown_connections 2023-11-15T04:57:41.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.487+0000 7f633f2cd700 1 --2- 172.21.15.178:0/3359652821 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f632402ffb0 0x7f6324032470 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:41.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.487+0000 7f633f2cd700 1 --2- 172.21.15.178:0/3359652821 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63380f6510 0x7f633806b9a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:41.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.487+0000 7f633f2cd700 1 -- 172.21.15.178:0/3359652821 >> 172.21.15.178:0/3359652821 conn(0x7f63380f1c70 msgr2=0x7f63380f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:41.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.487+0000 7f633f2cd700 1 -- 172.21.15.178:0/3359652821 shutdown_connections 2023-11-15T04:57:41.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:41.487+0000 7f633f2cd700 1 -- 172.21.15.178:0/3359652821 wait complete. 2023-11-15T04:57:41.491 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:42.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:41 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:40.329240+0000 mgr.smithi163.qesyrx (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:42.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:41 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:41.486553+0000 mon.smithi163 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.178:0/3359652821' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:42.872 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:42.872 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:57:44.064 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.062+0000 7f0020337700 1 -- 172.21.15.178:0/3879168118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00180f59d0 msgr2=0x7f00180f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:44.065 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.062+0000 7f0020337700 1 --2- 172.21.15.178:0/3879168118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00180f59d0 0x7f00180f5df0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f0008009fb0 tx=0x7f0008009b30).stop 2023-11-15T04:57:44.065 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.062+0000 7f0020337700 1 -- 172.21.15.178:0/3879168118 shutdown_connections 2023-11-15T04:57:44.066 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.062+0000 7f0020337700 1 --2- 172.21.15.178:0/3879168118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00180f59d0 0x7f00180f5df0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:44.066 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.062+0000 7f0020337700 1 -- 172.21.15.178:0/3879168118 >> 172.21.15.178:0/3879168118 conn(0x7f00180f02e0 msgr2=0x7f00180f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:44.066 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.062+0000 7f0020337700 1 -- 172.21.15.178:0/3879168118 shutdown_connections 2023-11-15T04:57:44.066 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.062+0000 7f0020337700 1 -- 172.21.15.178:0/3879168118 wait complete. 2023-11-15T04:57:44.067 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.063+0000 7f0020337700 1 Processor -- start 2023-11-15T04:57:44.067 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.063+0000 7f0020337700 1 -- start start 2023-11-15T04:57:44.067 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.063+0000 7f0020337700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00180f59d0 0x7f001810b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:44.067 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.063+0000 7f0020337700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00181071d0 con 0x7f00180f59d0 2023-11-15T04:57:44.067 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.063+0000 7f001e0d3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00180f59d0 0x7f001810b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:44.068 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.063+0000 7f001e0d3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00180f59d0 0x7f001810b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:58556/0 (socket says 172.21.15.178:58556) 2023-11-15T04:57:44.068 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.063+0000 7f001e0d3700 1 -- 172.21.15.178:0/1251611051 learned_addr learned my addr 172.21.15.178:0/1251611051 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:57:44.068 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.063+0000 7f001e0d3700 1 -- 172.21.15.178:0/1251611051 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0008005040 con 0x7f00180f59d0 2023-11-15T04:57:44.068 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.063+0000 7f001e0d3700 1 --2- 172.21.15.178:0/1251611051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00180f59d0 0x7f001810b7e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f0008013510 tx=0x7f000800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:44.070 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.064+0000 7f000effd700 1 -- 172.21.15.178:0/1251611051 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0008009320 con 0x7f00180f59d0 2023-11-15T04:57:44.071 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.064+0000 7f000effd700 1 -- 172.21.15.178:0/1251611051 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0008013f00 con 0x7f00180f59d0 2023-11-15T04:57:44.071 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.064+0000 7f0020337700 1 -- 172.21.15.178:0/1251611051 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00180f5390 con 0x7f00180f59d0 2023-11-15T04:57:44.071 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.064+0000 7f000effd700 1 -- 172.21.15.178:0/1251611051 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0008014d90 con 0x7f00180f59d0 2023-11-15T04:57:44.071 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.064+0000 7f0020337700 1 -- 172.21.15.178:0/1251611051 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f001810bcd0 con 0x7f00180f59d0 2023-11-15T04:57:44.071 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.064+0000 7f000effd700 1 -- 172.21.15.178:0/1251611051 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f00080263a0 con 0x7f00180f59d0 2023-11-15T04:57:44.072 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.065+0000 7f000effd700 1 --2- 172.21.15.178:0/1251611051 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f000402ff60 0x7f0004032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:44.072 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.065+0000 7f0020337700 1 -- 172.21.15.178:0/1251611051 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0018045ad0 con 0x7f00180f59d0 2023-11-15T04:57:44.072 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.065+0000 7f000effd700 1 -- 172.21.15.178:0/1251611051 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f00080410a0 con 0x7f00180f59d0 2023-11-15T04:57:44.072 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.065+0000 7f001d8d2700 1 --2- 172.21.15.178:0/1251611051 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f000402ff60 0x7f0004032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:44.072 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.065+0000 7f001d8d2700 1 --2- 172.21.15.178:0/1251611051 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f000402ff60 0x7f0004032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f00140096a0 tx=0x7f0014006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:44.073 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.070+0000 7f000effd700 1 -- 172.21.15.178:0/1251611051 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0008012920 con 0x7f00180f59d0 2023-11-15T04:57:44.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:43 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:42.329558+0000 mgr.smithi163.qesyrx (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:44.274 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.274+0000 7f0020337700 1 -- 172.21.15.178:0/1251611051 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f001810c490 con 0x7f00180f59d0 2023-11-15T04:57:44.275 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.275+0000 7f000effd700 1 -- 172.21.15.178:0/1251611051 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f000802b020 con 0x7f00180f59d0 2023-11-15T04:57:44.276 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:57:44.276 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:57:44.277 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.276+0000 7f0020337700 1 -- 172.21.15.178:0/1251611051 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f000402ff60 msgr2=0x7f0004032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:44.277 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.276+0000 7f0020337700 1 --2- 172.21.15.178:0/1251611051 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f000402ff60 0x7f0004032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f00140096a0 tx=0x7f0014006b40).stop 2023-11-15T04:57:44.277 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.277+0000 7f0020337700 1 -- 172.21.15.178:0/1251611051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00180f59d0 msgr2=0x7f001810b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:44.278 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.277+0000 7f0020337700 1 --2- 172.21.15.178:0/1251611051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00180f59d0 0x7f001810b7e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f0008013510 tx=0x7f000800b040).stop 2023-11-15T04:57:44.278 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.277+0000 7f0020337700 1 -- 172.21.15.178:0/1251611051 shutdown_connections 2023-11-15T04:57:44.278 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.277+0000 7f0020337700 1 --2- 172.21.15.178:0/1251611051 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f000402ff60 0x7f0004032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:44.278 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.277+0000 7f0020337700 1 --2- 172.21.15.178:0/1251611051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00180f59d0 0x7f001810b7e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:44.279 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.277+0000 7f0020337700 1 -- 172.21.15.178:0/1251611051 >> 172.21.15.178:0/1251611051 conn(0x7f00180f02e0 msgr2=0x7f00180f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:44.279 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.277+0000 7f0020337700 1 -- 172.21.15.178:0/1251611051 shutdown_connections 2023-11-15T04:57:44.279 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:44.277+0000 7f0020337700 1 -- 172.21.15.178:0/1251611051 wait complete. 2023-11-15T04:57:44.279 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:45.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:44 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:44.275567+0000 mon.smithi163 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.178:0/1251611051' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:45.651 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:45.652 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:57:46.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:45 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:44.329819+0000 mgr.smithi163.qesyrx (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:46.761 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.759+0000 7f45583df700 1 -- 172.21.15.178:0/577534082 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45500f5410 msgr2=0x7f45500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:46.762 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.759+0000 7f45583df700 1 --2- 172.21.15.178:0/577534082 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45500f5410 0x7f45500f5830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f4540009fb0 tx=0x7f4540009b30).stop 2023-11-15T04:57:46.762 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.759+0000 7f45583df700 1 -- 172.21.15.178:0/577534082 shutdown_connections 2023-11-15T04:57:46.762 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.759+0000 7f45583df700 1 --2- 172.21.15.178:0/577534082 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45500f5410 0x7f45500f5830 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:46.762 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.759+0000 7f45583df700 1 -- 172.21.15.178:0/577534082 >> 172.21.15.178:0/577534082 conn(0x7f45500f0b70 msgr2=0x7f45500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:46.762 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.759+0000 7f45583df700 1 -- 172.21.15.178:0/577534082 shutdown_connections 2023-11-15T04:57:46.763 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.759+0000 7f45583df700 1 -- 172.21.15.178:0/577534082 wait complete. 2023-11-15T04:57:46.763 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.760+0000 7f45583df700 1 Processor -- start 2023-11-15T04:57:46.763 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.760+0000 7f45583df700 1 -- start start 2023-11-15T04:57:46.763 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.760+0000 7f45583df700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45500f5410 0x7f4550109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:46.763 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.760+0000 7f45583df700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4550060220 con 0x7f45500f5410 2023-11-15T04:57:46.764 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.760+0000 7f455617b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45500f5410 0x7f4550109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:46.764 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.760+0000 7f455617b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45500f5410 0x7f4550109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:39314/0 (socket says 172.21.15.178:39314) 2023-11-15T04:57:46.764 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.760+0000 7f455617b700 1 -- 172.21.15.178:0/2808425099 learned_addr learned my addr 172.21.15.178:0/2808425099 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:57:46.764 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.761+0000 7f455617b700 1 -- 172.21.15.178:0/2808425099 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4540005040 con 0x7f45500f5410 2023-11-15T04:57:46.764 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.761+0000 7f455617b700 1 --2- 172.21.15.178:0/2808425099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45500f5410 0x7f4550109420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f4540013510 tx=0x7f454000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:46.765 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.761+0000 7f4546ffd700 1 -- 172.21.15.178:0/2808425099 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4540009320 con 0x7f45500f5410 2023-11-15T04:57:46.765 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.761+0000 7f45583df700 1 -- 172.21.15.178:0/2808425099 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4550060420 con 0x7f45500f5410 2023-11-15T04:57:46.765 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.761+0000 7f4546ffd700 1 -- 172.21.15.178:0/2808425099 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4540013f00 con 0x7f45500f5410 2023-11-15T04:57:46.765 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.761+0000 7f4546ffd700 1 -- 172.21.15.178:0/2808425099 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4540014d90 con 0x7f45500f5410 2023-11-15T04:57:46.765 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.761+0000 7f45583df700 1 -- 172.21.15.178:0/2808425099 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4550109ad0 con 0x7f45500f5410 2023-11-15T04:57:46.766 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.762+0000 7f4546ffd700 1 -- 172.21.15.178:0/2808425099 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45400095f0 con 0x7f45500f5410 2023-11-15T04:57:46.766 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.762+0000 7f45583df700 1 -- 172.21.15.178:0/2808425099 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4550059100 con 0x7f45500f5410 2023-11-15T04:57:46.767 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.762+0000 7f4546ffd700 1 --2- 172.21.15.178:0/2808425099 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f453c02ff60 0x7f453c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:46.767 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.762+0000 7f4546ffd700 1 -- 172.21.15.178:0/2808425099 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4540041000 con 0x7f45500f5410 2023-11-15T04:57:46.767 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.762+0000 7f455597a700 1 --2- 172.21.15.178:0/2808425099 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f453c02ff60 0x7f453c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:46.767 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.762+0000 7f455597a700 1 --2- 172.21.15.178:0/2808425099 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f453c02ff60 0x7f453c032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f454c0096a0 tx=0x7f454c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:46.768 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.767+0000 7f4546ffd700 1 -- 172.21.15.178:0/2808425099 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4540014420 con 0x7f45500f5410 2023-11-15T04:57:46.972 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.972+0000 7f45583df700 1 -- 172.21.15.178:0/2808425099 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4550045ad0 con 0x7f45500f5410 2023-11-15T04:57:46.973 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.972+0000 7f4546ffd700 1 -- 172.21.15.178:0/2808425099 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f454001d7f0 con 0x7f45500f5410 2023-11-15T04:57:46.973 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:57:46.974 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:57:46.975 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.974+0000 7f45583df700 1 -- 172.21.15.178:0/2808425099 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f453c02ff60 msgr2=0x7f453c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:46.975 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.974+0000 7f45583df700 1 --2- 172.21.15.178:0/2808425099 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f453c02ff60 0x7f453c032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f454c0096a0 tx=0x7f454c006b40).stop 2023-11-15T04:57:46.975 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.974+0000 7f45583df700 1 -- 172.21.15.178:0/2808425099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45500f5410 msgr2=0x7f4550109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:46.975 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.974+0000 7f45583df700 1 --2- 172.21.15.178:0/2808425099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45500f5410 0x7f4550109420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f4540013510 tx=0x7f454000b040).stop 2023-11-15T04:57:46.976 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.974+0000 7f45583df700 1 -- 172.21.15.178:0/2808425099 shutdown_connections 2023-11-15T04:57:46.976 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.974+0000 7f45583df700 1 --2- 172.21.15.178:0/2808425099 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f453c02ff60 0x7f453c032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:46.976 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.974+0000 7f45583df700 1 --2- 172.21.15.178:0/2808425099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45500f5410 0x7f4550109420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:46.976 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.974+0000 7f45583df700 1 -- 172.21.15.178:0/2808425099 >> 172.21.15.178:0/2808425099 conn(0x7f45500f0b70 msgr2=0x7f45500f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:46.976 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.974+0000 7f45583df700 1 -- 172.21.15.178:0/2808425099 shutdown_connections 2023-11-15T04:57:46.977 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:46.974+0000 7f45583df700 1 -- 172.21.15.178:0/2808425099 wait complete. 2023-11-15T04:57:46.977 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:48.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:47 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:46.330139+0000 mgr.smithi163.qesyrx (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:48.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:47 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:46.972980+0000 mon.smithi163 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.178:0/2808425099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:48.342 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:48.342 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:57:49.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.579+0000 7f6fa1e6a700 1 -- 172.21.15.178:0/2306956005 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f9c0601e0 msgr2=0x7f6f9c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:49.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.579+0000 7f6fa1e6a700 1 --2- 172.21.15.178:0/2306956005 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f9c0601e0 0x7f6f9c0f8f20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f6f90009fb0 tx=0x7f6f90009b30).stop 2023-11-15T04:57:49.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.579+0000 7f6fa1e6a700 1 -- 172.21.15.178:0/2306956005 shutdown_connections 2023-11-15T04:57:49.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.579+0000 7f6fa1e6a700 1 --2- 172.21.15.178:0/2306956005 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f9c0601e0 0x7f6f9c0f8f20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:49.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.579+0000 7f6fa1e6a700 1 -- 172.21.15.178:0/2306956005 >> 172.21.15.178:0/2306956005 conn(0x7f6f9c0f09a0 msgr2=0x7f6f9c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:49.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.579+0000 7f6fa1e6a700 1 -- 172.21.15.178:0/2306956005 shutdown_connections 2023-11-15T04:57:49.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.579+0000 7f6fa1e6a700 1 -- 172.21.15.178:0/2306956005 wait complete. 2023-11-15T04:57:49.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.579+0000 7f6fa1e6a700 1 Processor -- start 2023-11-15T04:57:49.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.579+0000 7f6fa1e6a700 1 -- start start 2023-11-15T04:57:49.584 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.580+0000 7f6fa1e6a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f9c0601e0 0x7f6f9c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:49.584 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.580+0000 7f6fa1e6a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c1069f0 con 0x7f6f9c0601e0 2023-11-15T04:57:49.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.580+0000 7f6fa0e68700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f9c0601e0 0x7f6f9c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:49.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.580+0000 7f6fa0e68700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f9c0601e0 0x7f6f9c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:39334/0 (socket says 172.21.15.178:39334) 2023-11-15T04:57:49.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.580+0000 7f6fa0e68700 1 -- 172.21.15.178:0/2536126044 learned_addr learned my addr 172.21.15.178:0/2536126044 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:57:49.586 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.581+0000 7f6fa0e68700 1 -- 172.21.15.178:0/2536126044 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f90005040 con 0x7f6f9c0601e0 2023-11-15T04:57:49.586 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.581+0000 7f6fa0e68700 1 --2- 172.21.15.178:0/2536126044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f9c0601e0 0x7f6f9c106500 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f6f90013510 tx=0x7f6f9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:49.586 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.581+0000 7f6f997fa700 1 -- 172.21.15.178:0/2536126044 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f90009320 con 0x7f6f9c0601e0 2023-11-15T04:57:49.587 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.581+0000 7f6f997fa700 1 -- 172.21.15.178:0/2536126044 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f90013f00 con 0x7f6f9c0601e0 2023-11-15T04:57:49.587 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.581+0000 7f6fa1e6a700 1 -- 172.21.15.178:0/2536126044 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f9c106bf0 con 0x7f6f9c0601e0 2023-11-15T04:57:49.588 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.581+0000 7f6f997fa700 1 -- 172.21.15.178:0/2536126044 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f90014d90 con 0x7f6f9c0601e0 2023-11-15T04:57:49.588 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.581+0000 7f6fa1e6a700 1 -- 172.21.15.178:0/2536126044 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f9c107080 con 0x7f6f9c0601e0 2023-11-15T04:57:49.588 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.582+0000 7f6f997fa700 1 -- 172.21.15.178:0/2536126044 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6f900095f0 con 0x7f6f9c0601e0 2023-11-15T04:57:49.589 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.582+0000 7f6f997fa700 1 --2- 172.21.15.178:0/2536126044 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6f8c02ff60 0x7f6f8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:49.589 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.582+0000 7f6fa1e6a700 1 -- 172.21.15.178:0/2536126044 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f9c045ad0 con 0x7f6f9c0601e0 2023-11-15T04:57:49.589 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.582+0000 7f6f997fa700 1 -- 172.21.15.178:0/2536126044 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f900410c0 con 0x7f6f9c0601e0 2023-11-15T04:57:49.590 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.582+0000 7f6f9bfff700 1 --2- 172.21.15.178:0/2536126044 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6f8c02ff60 0x7f6f8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:49.590 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.582+0000 7f6f9bfff700 1 --2- 172.21.15.178:0/2536126044 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6f8c02ff60 0x7f6f8c032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f6f880096a0 tx=0x7f6f88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:49.591 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.587+0000 7f6f997fa700 1 -- 172.21.15.178:0/2536126044 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f90014420 con 0x7f6f9c0601e0 2023-11-15T04:57:49.790 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.790+0000 7f6fa1e6a700 1 -- 172.21.15.178:0/2536126044 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f9c107f30 con 0x7f6f9c0601e0 2023-11-15T04:57:49.791 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.790+0000 7f6f997fa700 1 -- 172.21.15.178:0/2536126044 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6f9001d7f0 con 0x7f6f9c0601e0 2023-11-15T04:57:49.791 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:57:49.791 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:57:49.792 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.792+0000 7f6fa1e6a700 1 -- 172.21.15.178:0/2536126044 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6f8c02ff60 msgr2=0x7f6f8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:49.793 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.792+0000 7f6fa1e6a700 1 --2- 172.21.15.178:0/2536126044 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6f8c02ff60 0x7f6f8c032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f6f880096a0 tx=0x7f6f88006b40).stop 2023-11-15T04:57:49.793 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.792+0000 7f6fa1e6a700 1 -- 172.21.15.178:0/2536126044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f9c0601e0 msgr2=0x7f6f9c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:49.793 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.792+0000 7f6fa1e6a700 1 --2- 172.21.15.178:0/2536126044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f9c0601e0 0x7f6f9c106500 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f6f90013510 tx=0x7f6f9000b040).stop 2023-11-15T04:57:49.793 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.792+0000 7f6fa1e6a700 1 -- 172.21.15.178:0/2536126044 shutdown_connections 2023-11-15T04:57:49.793 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.792+0000 7f6fa1e6a700 1 --2- 172.21.15.178:0/2536126044 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6f8c02ff60 0x7f6f8c032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:49.794 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.792+0000 7f6fa1e6a700 1 --2- 172.21.15.178:0/2536126044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f9c0601e0 0x7f6f9c106500 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:49.794 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.792+0000 7f6fa1e6a700 1 -- 172.21.15.178:0/2536126044 >> 172.21.15.178:0/2536126044 conn(0x7f6f9c0f09a0 msgr2=0x7f6f9c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:49.794 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.792+0000 7f6fa1e6a700 1 -- 172.21.15.178:0/2536126044 shutdown_connections 2023-11-15T04:57:49.794 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:49.792+0000 7f6fa1e6a700 1 -- 172.21.15.178:0/2536126044 wait complete. 2023-11-15T04:57:49.795 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:50.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:49 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:48.330500+0000 mgr.smithi163.qesyrx (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:50.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:49 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:49.791181+0000 mon.smithi163 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.178:0/2536126044' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:51.176 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:51.177 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:57:52.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:51 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:50.330712+0000 mgr.smithi163.qesyrx (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:52.370 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.369+0000 7f18dac85700 1 -- 172.21.15.178:0/3805011998 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18d40f8700 msgr2=0x7f18d40f8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:52.371 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.369+0000 7f18dac85700 1 --2- 172.21.15.178:0/3805011998 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18d40f8700 0x7f18d40f8b20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f18c8009fb0 tx=0x7f18c8009b30).stop 2023-11-15T04:57:52.371 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.369+0000 7f18dac85700 1 -- 172.21.15.178:0/3805011998 shutdown_connections 2023-11-15T04:57:52.371 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.369+0000 7f18dac85700 1 --2- 172.21.15.178:0/3805011998 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18d40f8700 0x7f18d40f8b20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:52.371 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.369+0000 7f18dac85700 1 -- 172.21.15.178:0/3805011998 >> 172.21.15.178:0/3805011998 conn(0x7f18d40f40e0 msgr2=0x7f18d40f64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:52.372 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.369+0000 7f18dac85700 1 -- 172.21.15.178:0/3805011998 shutdown_connections 2023-11-15T04:57:52.372 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.369+0000 7f18dac85700 1 -- 172.21.15.178:0/3805011998 wait complete. 2023-11-15T04:57:52.372 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.370+0000 7f18dac85700 1 Processor -- start 2023-11-15T04:57:52.372 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.370+0000 7f18dac85700 1 -- start start 2023-11-15T04:57:52.372 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.370+0000 7f18dac85700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18d40f8700 0x7f18d4108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:52.373 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.370+0000 7f18dac85700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d4108e10 con 0x7f18d40f8700 2023-11-15T04:57:52.373 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.370+0000 7f18d8a21700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18d40f8700 0x7f18d4108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:52.373 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.370+0000 7f18d8a21700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18d40f8700 0x7f18d4108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:39340/0 (socket says 172.21.15.178:39340) 2023-11-15T04:57:52.373 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.370+0000 7f18d8a21700 1 -- 172.21.15.178:0/3468340660 learned_addr learned my addr 172.21.15.178:0/3468340660 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:57:52.375 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.371+0000 7f18d8a21700 1 -- 172.21.15.178:0/3468340660 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c8005040 con 0x7f18d40f8700 2023-11-15T04:57:52.375 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.371+0000 7f18d8a21700 1 --2- 172.21.15.178:0/3468340660 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18d40f8700 0x7f18d4108920 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f18c8013510 tx=0x7f18c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:52.375 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.371+0000 7f18d17fa700 1 -- 172.21.15.178:0/3468340660 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18c8009320 con 0x7f18d40f8700 2023-11-15T04:57:52.376 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.371+0000 7f18dac85700 1 -- 172.21.15.178:0/3468340660 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d4109010 con 0x7f18d40f8700 2023-11-15T04:57:52.376 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.371+0000 7f18d17fa700 1 -- 172.21.15.178:0/3468340660 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f18c8013f00 con 0x7f18d40f8700 2023-11-15T04:57:52.376 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.371+0000 7f18d17fa700 1 -- 172.21.15.178:0/3468340660 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18c8014d90 con 0x7f18d40f8700 2023-11-15T04:57:52.376 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.371+0000 7f18dac85700 1 -- 172.21.15.178:0/3468340660 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d41094a0 con 0x7f18d40f8700 2023-11-15T04:57:52.376 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.372+0000 7f18d17fa700 1 -- 172.21.15.178:0/3468340660 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f18c80263a0 con 0x7f18d40f8700 2023-11-15T04:57:52.376 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.372+0000 7f18dac85700 1 -- 172.21.15.178:0/3468340660 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d4045ad0 con 0x7f18d40f8700 2023-11-15T04:57:52.377 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.372+0000 7f18d17fa700 1 --2- 172.21.15.178:0/3468340660 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f18c402ff60 0x7f18c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:52.377 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.372+0000 7f18d17fa700 1 -- 172.21.15.178:0/3468340660 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f18c80413c0 con 0x7f18d40f8700 2023-11-15T04:57:52.377 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.372+0000 7f18d3fff700 1 --2- 172.21.15.178:0/3468340660 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f18c402ff60 0x7f18c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:52.377 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.373+0000 7f18d3fff700 1 --2- 172.21.15.178:0/3468340660 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f18c402ff60 0x7f18c4032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f18c00096a0 tx=0x7f18c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:52.378 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.377+0000 7f18d17fa700 1 -- 172.21.15.178:0/3468340660 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f18c80126f0 con 0x7f18d40f8700 2023-11-15T04:57:52.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.581+0000 7f18dac85700 1 -- 172.21.15.178:0/3468340660 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18d40fe150 con 0x7f18d40f8700 2023-11-15T04:57:52.582 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.582+0000 7f18d17fa700 1 -- 172.21.15.178:0/3468340660 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f18c8012330 con 0x7f18d40f8700 2023-11-15T04:57:52.582 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:57:52.582 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:57:52.583 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.583+0000 7f18dac85700 1 -- 172.21.15.178:0/3468340660 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f18c402ff60 msgr2=0x7f18c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:52.584 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.583+0000 7f18dac85700 1 --2- 172.21.15.178:0/3468340660 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f18c402ff60 0x7f18c4032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f18c00096a0 tx=0x7f18c0006b40).stop 2023-11-15T04:57:52.584 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.583+0000 7f18dac85700 1 -- 172.21.15.178:0/3468340660 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18d40f8700 msgr2=0x7f18d4108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:52.584 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.583+0000 7f18dac85700 1 --2- 172.21.15.178:0/3468340660 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18d40f8700 0x7f18d4108920 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f18c8013510 tx=0x7f18c800b040).stop 2023-11-15T04:57:52.584 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.583+0000 7f18dac85700 1 -- 172.21.15.178:0/3468340660 shutdown_connections 2023-11-15T04:57:52.584 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.583+0000 7f18dac85700 1 --2- 172.21.15.178:0/3468340660 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f18c402ff60 0x7f18c4032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:52.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.583+0000 7f18dac85700 1 --2- 172.21.15.178:0/3468340660 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18d40f8700 0x7f18d4108920 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:52.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.583+0000 7f18dac85700 1 -- 172.21.15.178:0/3468340660 >> 172.21.15.178:0/3468340660 conn(0x7f18d40f40e0 msgr2=0x7f18d40f5be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:52.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.583+0000 7f18dac85700 1 -- 172.21.15.178:0/3468340660 shutdown_connections 2023-11-15T04:57:52.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:52.583+0000 7f18dac85700 1 -- 172.21.15.178:0/3468340660 wait complete. 2023-11-15T04:57:52.586 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:53.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:52 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:52.582282+0000 mon.smithi163 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.178:0/3468340660' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:53.960 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:53.961 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:57:54.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:53 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:52.330972+0000 mgr.smithi163.qesyrx (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:55.124 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.124+0000 7f669cdea700 1 -- 172.21.15.178:0/1974243727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66980f51a0 msgr2=0x7f66980f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:55.125 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.124+0000 7f669cdea700 1 --2- 172.21.15.178:0/1974243727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66980f51a0 0x7f66980f55c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f6688009fb0 tx=0x7f6688009b30).stop 2023-11-15T04:57:55.125 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.124+0000 7f669cdea700 1 -- 172.21.15.178:0/1974243727 shutdown_connections 2023-11-15T04:57:55.125 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.124+0000 7f669cdea700 1 --2- 172.21.15.178:0/1974243727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66980f51a0 0x7f66980f55c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:55.126 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.124+0000 7f669cdea700 1 -- 172.21.15.178:0/1974243727 >> 172.21.15.178:0/1974243727 conn(0x7f66980f09a0 msgr2=0x7f66980f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:55.126 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.124+0000 7f669cdea700 1 -- 172.21.15.178:0/1974243727 shutdown_connections 2023-11-15T04:57:55.126 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.124+0000 7f669cdea700 1 -- 172.21.15.178:0/1974243727 wait complete. 2023-11-15T04:57:55.126 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.124+0000 7f669cdea700 1 Processor -- start 2023-11-15T04:57:55.128 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.124+0000 7f669cdea700 1 -- start start 2023-11-15T04:57:55.129 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.125+0000 7f669cdea700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66980f51a0 0x7f66981064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:55.129 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.125+0000 7f669cdea700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66981069b0 con 0x7f66980f51a0 2023-11-15T04:57:55.129 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.125+0000 7f66977fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66980f51a0 0x7f66981064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:55.129 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.125+0000 7f66977fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66980f51a0 0x7f66981064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:35810/0 (socket says 172.21.15.178:35810) 2023-11-15T04:57:55.129 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.125+0000 7f66977fe700 1 -- 172.21.15.178:0/2404536748 learned_addr learned my addr 172.21.15.178:0/2404536748 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:57:55.130 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.125+0000 7f66977fe700 1 -- 172.21.15.178:0/2404536748 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6688005040 con 0x7f66980f51a0 2023-11-15T04:57:55.130 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.125+0000 7f66977fe700 1 --2- 172.21.15.178:0/2404536748 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66980f51a0 0x7f66981064c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f6688013510 tx=0x7f668800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:55.130 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.125+0000 7f667ffff700 1 -- 172.21.15.178:0/2404536748 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6688009320 con 0x7f66980f51a0 2023-11-15T04:57:55.130 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.125+0000 7f669cdea700 1 -- 172.21.15.178:0/2404536748 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6698106bb0 con 0x7f66980f51a0 2023-11-15T04:57:55.131 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.125+0000 7f667ffff700 1 -- 172.21.15.178:0/2404536748 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6688013f00 con 0x7f66980f51a0 2023-11-15T04:57:55.131 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.125+0000 7f667ffff700 1 -- 172.21.15.178:0/2404536748 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6688014dc0 con 0x7f66980f51a0 2023-11-15T04:57:55.131 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.126+0000 7f669cdea700 1 -- 172.21.15.178:0/2404536748 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6698107040 con 0x7f66980f51a0 2023-11-15T04:57:55.131 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.126+0000 7f667ffff700 1 -- 172.21.15.178:0/2404536748 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6688014420 con 0x7f66980f51a0 2023-11-15T04:57:55.132 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.126+0000 7f669cdea700 1 -- 172.21.15.178:0/2404536748 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6698045ad0 con 0x7f66980f51a0 2023-11-15T04:57:55.132 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.126+0000 7f667ffff700 1 --2- 172.21.15.178:0/2404536748 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f668002ff60 0x7f6680032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:55.132 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.126+0000 7f667ffff700 1 -- 172.21.15.178:0/2404536748 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6688041050 con 0x7f66980f51a0 2023-11-15T04:57:55.133 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.127+0000 7f6696ffd700 1 --2- 172.21.15.178:0/2404536748 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f668002ff60 0x7f6680032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:55.133 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.127+0000 7f6696ffd700 1 --2- 172.21.15.178:0/2404536748 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f668002ff60 0x7f6680032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f668c0096a0 tx=0x7f668c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:55.133 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.131+0000 7f667ffff700 1 -- 172.21.15.178:0/2404536748 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66880095f0 con 0x7f66980f51a0 2023-11-15T04:57:55.338 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.337+0000 7f669cdea700 1 -- 172.21.15.178:0/2404536748 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6698107ef0 con 0x7f66980f51a0 2023-11-15T04:57:55.339 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.338+0000 7f667ffff700 1 -- 172.21.15.178:0/2404536748 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f668801cb10 con 0x7f66980f51a0 2023-11-15T04:57:55.339 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:57:55.339 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:57:55.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.340+0000 7f669cdea700 1 -- 172.21.15.178:0/2404536748 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f668002ff60 msgr2=0x7f6680032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:55.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.340+0000 7f669cdea700 1 --2- 172.21.15.178:0/2404536748 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f668002ff60 0x7f6680032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f668c0096a0 tx=0x7f668c006b40).stop 2023-11-15T04:57:55.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.340+0000 7f669cdea700 1 -- 172.21.15.178:0/2404536748 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66980f51a0 msgr2=0x7f66981064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:55.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.340+0000 7f669cdea700 1 --2- 172.21.15.178:0/2404536748 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66980f51a0 0x7f66981064c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f6688013510 tx=0x7f668800b040).stop 2023-11-15T04:57:55.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.340+0000 7f669cdea700 1 -- 172.21.15.178:0/2404536748 shutdown_connections 2023-11-15T04:57:55.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.340+0000 7f669cdea700 1 --2- 172.21.15.178:0/2404536748 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f668002ff60 0x7f6680032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:55.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.340+0000 7f669cdea700 1 --2- 172.21.15.178:0/2404536748 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66980f51a0 0x7f66981064c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:55.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.340+0000 7f669cdea700 1 -- 172.21.15.178:0/2404536748 >> 172.21.15.178:0/2404536748 conn(0x7f66980f09a0 msgr2=0x7f66980f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:55.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.340+0000 7f669cdea700 1 -- 172.21.15.178:0/2404536748 shutdown_connections 2023-11-15T04:57:55.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:55.340+0000 7f669cdea700 1 -- 172.21.15.178:0/2404536748 wait complete. 2023-11-15T04:57:55.343 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:56.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:55 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:54.331334+0000 mgr.smithi163.qesyrx (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:56.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:55 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:55.338872+0000 mon.smithi163 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.178:0/2404536748' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:56.714 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:56.715 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:57:57.894 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.893+0000 7f6bb3e66700 1 -- 172.21.15.178:0/2279491238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bac060280 msgr2=0x7f6bac0fb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:57.895 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.893+0000 7f6bb3e66700 1 --2- 172.21.15.178:0/2279491238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bac060280 0x7f6bac0fb4f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f6b9c009fb0 tx=0x7f6b9c009b30).stop 2023-11-15T04:57:57.895 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.893+0000 7f6bb3e66700 1 -- 172.21.15.178:0/2279491238 shutdown_connections 2023-11-15T04:57:57.895 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.893+0000 7f6bb3e66700 1 --2- 172.21.15.178:0/2279491238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bac060280 0x7f6bac0fb4f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:57.895 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.893+0000 7f6bb3e66700 1 -- 172.21.15.178:0/2279491238 >> 172.21.15.178:0/2279491238 conn(0x7f6bac0f0b70 msgr2=0x7f6bac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:57.895 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.893+0000 7f6bb3e66700 1 -- 172.21.15.178:0/2279491238 shutdown_connections 2023-11-15T04:57:57.896 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.893+0000 7f6bb3e66700 1 -- 172.21.15.178:0/2279491238 wait complete. 2023-11-15T04:57:57.896 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.894+0000 7f6bb3e66700 1 Processor -- start 2023-11-15T04:57:57.896 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.894+0000 7f6bb3e66700 1 -- start start 2023-11-15T04:57:57.898 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.894+0000 7f6bb3e66700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bac060280 0x7f6bac1088e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:57.898 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.894+0000 7f6bb3e66700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac108dd0 con 0x7f6bac060280 2023-11-15T04:57:57.898 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.895+0000 7f6bb1c02700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bac060280 0x7f6bac1088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:57.899 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.895+0000 7f6bb1c02700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bac060280 0x7f6bac1088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:35830/0 (socket says 172.21.15.178:35830) 2023-11-15T04:57:57.899 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.895+0000 7f6bb1c02700 1 -- 172.21.15.178:0/38840731 learned_addr learned my addr 172.21.15.178:0/38840731 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:57:57.899 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.895+0000 7f6bb1c02700 1 -- 172.21.15.178:0/38840731 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9c005040 con 0x7f6bac060280 2023-11-15T04:57:57.899 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.895+0000 7f6bb1c02700 1 --2- 172.21.15.178:0/38840731 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bac060280 0x7f6bac1088e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f6b9c013510 tx=0x7f6b9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:57.899 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.895+0000 7f6ba27fc700 1 -- 172.21.15.178:0/38840731 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b9c009320 con 0x7f6bac060280 2023-11-15T04:57:57.900 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.895+0000 7f6bb3e66700 1 -- 172.21.15.178:0/38840731 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bac108fd0 con 0x7f6bac060280 2023-11-15T04:57:57.900 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.895+0000 7f6ba27fc700 1 -- 172.21.15.178:0/38840731 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b9c013f00 con 0x7f6bac060280 2023-11-15T04:57:57.900 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.895+0000 7f6ba27fc700 1 -- 172.21.15.178:0/38840731 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b9c014dc0 con 0x7f6bac060280 2023-11-15T04:57:57.900 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.895+0000 7f6bb3e66700 1 -- 172.21.15.178:0/38840731 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bac109460 con 0x7f6bac060280 2023-11-15T04:57:57.900 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.896+0000 7f6ba27fc700 1 -- 172.21.15.178:0/38840731 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6b9c014420 con 0x7f6bac060280 2023-11-15T04:57:57.901 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.896+0000 7f6bb3e66700 1 -- 172.21.15.178:0/38840731 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bac045ad0 con 0x7f6bac060280 2023-11-15T04:57:57.901 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.896+0000 7f6ba27fc700 1 --2- 172.21.15.178:0/38840731 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6b9802ff60 0x7f6b98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:57:57.901 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.896+0000 7f6ba27fc700 1 -- 172.21.15.178:0/38840731 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6b9c041110 con 0x7f6bac060280 2023-11-15T04:57:57.901 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.896+0000 7f6bb1401700 1 --2- 172.21.15.178:0/38840731 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6b9802ff60 0x7f6b98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:57:57.901 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.897+0000 7f6bb1401700 1 --2- 172.21.15.178:0/38840731 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6b9802ff60 0x7f6b98032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f6ba80096a0 tx=0x7f6ba8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:57:57.902 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:57.901+0000 7f6ba27fc700 1 -- 172.21.15.178:0/38840731 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b9c0095f0 con 0x7f6bac060280 2023-11-15T04:57:58.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:58.107+0000 7f6bb3e66700 1 -- 172.21.15.178:0/38840731 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6bac10a310 con 0x7f6bac060280 2023-11-15T04:57:58.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:58.107+0000 7f6ba27fc700 1 -- 172.21.15.178:0/38840731 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6b9c01cb10 con 0x7f6bac060280 2023-11-15T04:57:58.108 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:57:58.108 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:57:58.110 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:58.109+0000 7f6bb3e66700 1 -- 172.21.15.178:0/38840731 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6b9802ff60 msgr2=0x7f6b98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:58.110 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:58.109+0000 7f6bb3e66700 1 --2- 172.21.15.178:0/38840731 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6b9802ff60 0x7f6b98032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f6ba80096a0 tx=0x7f6ba8006b40).stop 2023-11-15T04:57:58.110 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:58.109+0000 7f6bb3e66700 1 -- 172.21.15.178:0/38840731 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bac060280 msgr2=0x7f6bac1088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:57:58.110 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:58.109+0000 7f6bb3e66700 1 --2- 172.21.15.178:0/38840731 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bac060280 0x7f6bac1088e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f6b9c013510 tx=0x7f6b9c00b040).stop 2023-11-15T04:57:58.111 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:58.109+0000 7f6bb3e66700 1 -- 172.21.15.178:0/38840731 shutdown_connections 2023-11-15T04:57:58.111 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:58.109+0000 7f6bb3e66700 1 --2- 172.21.15.178:0/38840731 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6b9802ff60 0x7f6b98032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:58.111 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:58.109+0000 7f6bb3e66700 1 --2- 172.21.15.178:0/38840731 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bac060280 0x7f6bac1088e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:57:58.111 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:58.109+0000 7f6bb3e66700 1 -- 172.21.15.178:0/38840731 >> 172.21.15.178:0/38840731 conn(0x7f6bac0f0b70 msgr2=0x7f6bac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:57:58.111 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:58.109+0000 7f6bb3e66700 1 -- 172.21.15.178:0/38840731 shutdown_connections 2023-11-15T04:57:58.112 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:57:58.109+0000 7f6bb3e66700 1 -- 172.21.15.178:0/38840731 wait complete. 2023-11-15T04:57:58.112 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:57:58.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:57 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:56.331738+0000 mgr.smithi163.qesyrx (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:57:59.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:58 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:57:58.108074+0000 mon.smithi163 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.178:0/38840731' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:57:59.453 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:57:59.454 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:00.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:57:59 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:57:58.331978+0000 mgr.smithi163.qesyrx (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:00.579 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.576+0000 7ff3f8f89700 1 -- 172.21.15.178:0/2534973411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3f40f6510 msgr2=0x7ff3f40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:00.579 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.576+0000 7ff3f8f89700 1 --2- 172.21.15.178:0/2534973411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3f40f6510 0x7ff3f40f6930 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7ff3e4009960 tx=0x7ff3e4004d10).stop 2023-11-15T04:58:00.579 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.577+0000 7ff3f8f89700 1 -- 172.21.15.178:0/2534973411 shutdown_connections 2023-11-15T04:58:00.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.577+0000 7ff3f8f89700 1 --2- 172.21.15.178:0/2534973411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3f40f6510 0x7ff3f40f6930 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:00.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.577+0000 7ff3f8f89700 1 -- 172.21.15.178:0/2534973411 >> 172.21.15.178:0/2534973411 conn(0x7ff3f40f1c70 msgr2=0x7ff3f40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:00.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.577+0000 7ff3f8f89700 1 -- 172.21.15.178:0/2534973411 shutdown_connections 2023-11-15T04:58:00.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.577+0000 7ff3f8f89700 1 -- 172.21.15.178:0/2534973411 wait complete. 2023-11-15T04:58:00.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.577+0000 7ff3f8f89700 1 Processor -- start 2023-11-15T04:58:00.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.577+0000 7ff3f8f89700 1 -- start start 2023-11-15T04:58:00.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.577+0000 7ff3f8f89700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3f40f6510 0x7ff3f4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:00.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.577+0000 7ff3f8f89700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f4106c20 con 0x7ff3f40f6510 2023-11-15T04:58:00.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.577+0000 7ff3f259c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3f40f6510 0x7ff3f4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:00.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.577+0000 7ff3f259c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3f40f6510 0x7ff3f4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:35844/0 (socket says 172.21.15.178:35844) 2023-11-15T04:58:00.582 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.577+0000 7ff3f259c700 1 -- 172.21.15.178:0/3142384362 learned_addr learned my addr 172.21.15.178:0/3142384362 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:00.582 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.578+0000 7ff3f259c700 1 -- 172.21.15.178:0/3142384362 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e4009640 con 0x7ff3f40f6510 2023-11-15T04:58:00.582 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.578+0000 7ff3f259c700 1 --2- 172.21.15.178:0/3142384362 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3f40f6510 0x7ff3f4106730 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7ff3e4004b70 tx=0x7ff3e4004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:00.582 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.578+0000 7ff3eaffd700 1 -- 172.21.15.178:0/3142384362 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3e4018070 con 0x7ff3f40f6510 2023-11-15T04:58:00.582 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.578+0000 7ff3f8f89700 1 -- 172.21.15.178:0/3142384362 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f4106e20 con 0x7ff3f40f6510 2023-11-15T04:58:00.583 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.578+0000 7ff3eaffd700 1 -- 172.21.15.178:0/3142384362 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3e400b870 con 0x7ff3f40f6510 2023-11-15T04:58:00.583 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.578+0000 7ff3eaffd700 1 -- 172.21.15.178:0/3142384362 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3e401c560 con 0x7ff3f40f6510 2023-11-15T04:58:00.583 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.578+0000 7ff3f8f89700 1 -- 172.21.15.178:0/3142384362 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f41072b0 con 0x7ff3f40f6510 2023-11-15T04:58:00.584 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.579+0000 7ff3f8f89700 1 -- 172.21.15.178:0/3142384362 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f4045ad0 con 0x7ff3f40f6510 2023-11-15T04:58:00.584 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.579+0000 7ff3eaffd700 1 -- 172.21.15.178:0/3142384362 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3e4013630 con 0x7ff3f40f6510 2023-11-15T04:58:00.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.579+0000 7ff3eaffd700 1 --2- 172.21.15.178:0/3142384362 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff3e002ff60 0x7ff3e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:00.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.579+0000 7ff3eaffd700 1 -- 172.21.15.178:0/3142384362 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff3e40411d0 con 0x7ff3f40f6510 2023-11-15T04:58:00.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.579+0000 7ff3f1d9b700 1 --2- 172.21.15.178:0/3142384362 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff3e002ff60 0x7ff3e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:00.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.580+0000 7ff3f1d9b700 1 --2- 172.21.15.178:0/3142384362 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff3e002ff60 0x7ff3e0032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7ff3dc0096a0 tx=0x7ff3dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:00.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.584+0000 7ff3eaffd700 1 -- 172.21.15.178:0/3142384362 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3e4004810 con 0x7ff3f40f6510 2023-11-15T04:58:00.792 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.792+0000 7ff3f8f89700 1 -- 172.21.15.178:0/3142384362 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3f40fbf60 con 0x7ff3f40f6510 2023-11-15T04:58:00.793 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.793+0000 7ff3eaffd700 1 -- 172.21.15.178:0/3142384362 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff3e4004e10 con 0x7ff3f40f6510 2023-11-15T04:58:00.794 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:00.794 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:00.795 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.794+0000 7ff3f8f89700 1 -- 172.21.15.178:0/3142384362 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff3e002ff60 msgr2=0x7ff3e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:00.796 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.794+0000 7ff3f8f89700 1 --2- 172.21.15.178:0/3142384362 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff3e002ff60 0x7ff3e0032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7ff3dc0096a0 tx=0x7ff3dc006b40).stop 2023-11-15T04:58:00.796 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.794+0000 7ff3f8f89700 1 -- 172.21.15.178:0/3142384362 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3f40f6510 msgr2=0x7ff3f4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:00.797 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.794+0000 7ff3f8f89700 1 --2- 172.21.15.178:0/3142384362 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3f40f6510 0x7ff3f4106730 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7ff3e4004b70 tx=0x7ff3e4004d10).stop 2023-11-15T04:58:00.797 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.795+0000 7ff3f8f89700 1 -- 172.21.15.178:0/3142384362 shutdown_connections 2023-11-15T04:58:00.797 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.795+0000 7ff3f8f89700 1 --2- 172.21.15.178:0/3142384362 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff3e002ff60 0x7ff3e0032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:00.798 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.795+0000 7ff3f8f89700 1 --2- 172.21.15.178:0/3142384362 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3f40f6510 0x7ff3f4106730 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:00.798 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.795+0000 7ff3f8f89700 1 -- 172.21.15.178:0/3142384362 >> 172.21.15.178:0/3142384362 conn(0x7ff3f40f1c70 msgr2=0x7ff3f40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:00.798 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.795+0000 7ff3f8f89700 1 -- 172.21.15.178:0/3142384362 shutdown_connections 2023-11-15T04:58:00.799 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:00.795+0000 7ff3f8f89700 1 -- 172.21.15.178:0/3142384362 wait complete. 2023-11-15T04:58:00.799 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:58:01.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:00 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:00.793495+0000 mon.smithi163 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.178:0/3142384362' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:58:02.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:01 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:00.332330+0000 mgr.smithi163.qesyrx (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:02.220 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:58:02.221 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:03.400 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.398+0000 7f1fad557700 1 -- 172.21.15.178:0/542091758 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fa80f62a0 msgr2=0x7f1fa80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:03.400 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.398+0000 7f1fad557700 1 --2- 172.21.15.178:0/542091758 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fa80f62a0 0x7f1fa80f66c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f1f98009fb0 tx=0x7f1f98009b30).stop 2023-11-15T04:58:03.400 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.398+0000 7f1fad557700 1 -- 172.21.15.178:0/542091758 shutdown_connections 2023-11-15T04:58:03.401 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.398+0000 7f1fad557700 1 --2- 172.21.15.178:0/542091758 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fa80f62a0 0x7f1fa80f66c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:03.401 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.398+0000 7f1fad557700 1 -- 172.21.15.178:0/542091758 >> 172.21.15.178:0/542091758 conn(0x7f1fa80f1aa0 msgr2=0x7f1fa80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:03.401 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.398+0000 7f1fad557700 1 -- 172.21.15.178:0/542091758 shutdown_connections 2023-11-15T04:58:03.401 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.398+0000 7f1fad557700 1 -- 172.21.15.178:0/542091758 wait complete. 2023-11-15T04:58:03.402 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.398+0000 7f1fad557700 1 Processor -- start 2023-11-15T04:58:03.402 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.399+0000 7f1fad557700 1 -- start start 2023-11-15T04:58:03.402 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.399+0000 7f1fad557700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fa80f62a0 0x7f1fa81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:03.402 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.399+0000 7f1fad557700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa81069a0 con 0x7f1fa80f62a0 2023-11-15T04:58:03.402 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.399+0000 7f1fa7fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fa80f62a0 0x7f1fa81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:03.403 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.399+0000 7f1fa7fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fa80f62a0 0x7f1fa81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:35870/0 (socket says 172.21.15.178:35870) 2023-11-15T04:58:03.403 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.399+0000 7f1fa7fff700 1 -- 172.21.15.178:0/276557570 learned_addr learned my addr 172.21.15.178:0/276557570 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:03.403 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.399+0000 7f1fa7fff700 1 -- 172.21.15.178:0/276557570 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f98005040 con 0x7f1fa80f62a0 2023-11-15T04:58:03.403 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.399+0000 7f1fa7fff700 1 --2- 172.21.15.178:0/276557570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fa80f62a0 0x7f1fa81064b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f1f98013510 tx=0x7f1f9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:03.403 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.400+0000 7f1fa4ff9700 1 -- 172.21.15.178:0/276557570 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f98009320 con 0x7f1fa80f62a0 2023-11-15T04:58:03.404 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.400+0000 7f1fad557700 1 -- 172.21.15.178:0/276557570 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fa8106ba0 con 0x7f1fa80f62a0 2023-11-15T04:58:03.404 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.400+0000 7f1fa4ff9700 1 -- 172.21.15.178:0/276557570 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f98013f00 con 0x7f1fa80f62a0 2023-11-15T04:58:03.404 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.400+0000 7f1fa4ff9700 1 -- 172.21.15.178:0/276557570 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f98014dc0 con 0x7f1fa80f62a0 2023-11-15T04:58:03.404 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.400+0000 7f1fad557700 1 -- 172.21.15.178:0/276557570 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fa8107030 con 0x7f1fa80f62a0 2023-11-15T04:58:03.405 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.400+0000 7f1fa4ff9700 1 -- 172.21.15.178:0/276557570 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1f980095f0 con 0x7f1fa80f62a0 2023-11-15T04:58:03.405 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.400+0000 7f1fad557700 1 -- 172.21.15.178:0/276557570 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa8045ad0 con 0x7f1fa80f62a0 2023-11-15T04:58:03.406 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.401+0000 7f1fa4ff9700 1 --2- 172.21.15.178:0/276557570 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1f9002ff60 0x7f1f90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:03.406 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.401+0000 7f1fa4ff9700 1 -- 172.21.15.178:0/276557570 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1f980410a0 con 0x7f1fa80f62a0 2023-11-15T04:58:03.406 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.401+0000 7f1fa77fe700 1 --2- 172.21.15.178:0/276557570 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1f9002ff60 0x7f1f90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:03.406 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.401+0000 7f1fa77fe700 1 --2- 172.21.15.178:0/276557570 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1f9002ff60 0x7f1f90032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f1f9c0096a0 tx=0x7f1f9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:03.407 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.405+0000 7f1fa4ff9700 1 -- 172.21.15.178:0/276557570 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f98014420 con 0x7f1fa80f62a0 2023-11-15T04:58:03.612 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.612+0000 7f1fad557700 1 -- 172.21.15.178:0/276557570 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1fa8107e80 con 0x7f1fa80f62a0 2023-11-15T04:58:03.613 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.612+0000 7f1fa4ff9700 1 -- 172.21.15.178:0/276557570 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1f9801c920 con 0x7f1fa80f62a0 2023-11-15T04:58:03.613 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:03.613 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:03.615 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.614+0000 7f1fad557700 1 -- 172.21.15.178:0/276557570 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1f9002ff60 msgr2=0x7f1f90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:03.615 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.614+0000 7f1fad557700 1 --2- 172.21.15.178:0/276557570 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1f9002ff60 0x7f1f90032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f1f9c0096a0 tx=0x7f1f9c006b40).stop 2023-11-15T04:58:03.615 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.614+0000 7f1fad557700 1 -- 172.21.15.178:0/276557570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fa80f62a0 msgr2=0x7f1fa81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:03.615 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.614+0000 7f1fad557700 1 --2- 172.21.15.178:0/276557570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fa80f62a0 0x7f1fa81064b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f1f98013510 tx=0x7f1f9800b040).stop 2023-11-15T04:58:03.615 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.614+0000 7f1fad557700 1 -- 172.21.15.178:0/276557570 shutdown_connections 2023-11-15T04:58:03.616 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.614+0000 7f1fad557700 1 --2- 172.21.15.178:0/276557570 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1f9002ff60 0x7f1f90032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:03.616 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.614+0000 7f1fad557700 1 --2- 172.21.15.178:0/276557570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fa80f62a0 0x7f1fa81064b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:03.616 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.614+0000 7f1fad557700 1 -- 172.21.15.178:0/276557570 >> 172.21.15.178:0/276557570 conn(0x7f1fa80f1aa0 msgr2=0x7f1fa80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:03.616 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.614+0000 7f1fad557700 1 -- 172.21.15.178:0/276557570 shutdown_connections 2023-11-15T04:58:03.616 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:03.614+0000 7f1fad557700 1 -- 172.21.15.178:0/276557570 wait complete. 2023-11-15T04:58:03.617 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:58:04.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:03 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:02.332663+0000 mgr.smithi163.qesyrx (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:04.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:03 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:03.613226+0000 mon.smithi163 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.178:0/276557570' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:58:04.970 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:58:04.970 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:06.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:05 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:04.332985+0000 mgr.smithi163.qesyrx (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:06.205 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.204+0000 7f9cd3618700 1 -- 172.21.15.178:0/1772499757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ccc060220 msgr2=0x7f9ccc0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:06.206 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.204+0000 7f9cd3618700 1 --2- 172.21.15.178:0/1772499757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ccc060220 0x7f9ccc0f9150 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f9cc0009fb0 tx=0x7f9cc0009b30).stop 2023-11-15T04:58:06.206 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.204+0000 7f9cd3618700 1 -- 172.21.15.178:0/1772499757 shutdown_connections 2023-11-15T04:58:06.206 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.204+0000 7f9cd3618700 1 --2- 172.21.15.178:0/1772499757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ccc060220 0x7f9ccc0f9150 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:06.206 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.204+0000 7f9cd3618700 1 -- 172.21.15.178:0/1772499757 >> 172.21.15.178:0/1772499757 conn(0x7f9ccc0f0b70 msgr2=0x7f9ccc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:06.206 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.204+0000 7f9cd3618700 1 -- 172.21.15.178:0/1772499757 shutdown_connections 2023-11-15T04:58:06.207 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.204+0000 7f9cd3618700 1 -- 172.21.15.178:0/1772499757 wait complete. 2023-11-15T04:58:06.207 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.205+0000 7f9cd3618700 1 Processor -- start 2023-11-15T04:58:06.207 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.205+0000 7f9cd3618700 1 -- start start 2023-11-15T04:58:06.207 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.205+0000 7f9cd3618700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ccc060220 0x7f9ccc0f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:06.207 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.205+0000 7f9cd3618700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ccc0f5050 con 0x7f9ccc060220 2023-11-15T04:58:06.208 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.205+0000 7f9cd13b4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ccc060220 0x7f9ccc0f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:06.208 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.205+0000 7f9cd13b4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ccc060220 0x7f9ccc0f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:41488/0 (socket says 172.21.15.178:41488) 2023-11-15T04:58:06.208 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.205+0000 7f9cd13b4700 1 -- 172.21.15.178:0/682607788 learned_addr learned my addr 172.21.15.178:0/682607788 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:06.209 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.206+0000 7f9cd13b4700 1 -- 172.21.15.178:0/682607788 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc0005040 con 0x7f9ccc060220 2023-11-15T04:58:06.210 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.206+0000 7f9cd13b4700 1 --2- 172.21.15.178:0/682607788 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ccc060220 0x7f9ccc0f6bd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f9cc0013510 tx=0x7f9cc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:06.210 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.206+0000 7f9cbdffb700 1 -- 172.21.15.178:0/682607788 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cc0009320 con 0x7f9ccc060220 2023-11-15T04:58:06.210 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.206+0000 7f9cbdffb700 1 -- 172.21.15.178:0/682607788 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9cc0013f00 con 0x7f9ccc060220 2023-11-15T04:58:06.210 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.206+0000 7f9cd3618700 1 -- 172.21.15.178:0/682607788 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ccc0f51f0 con 0x7f9ccc060220 2023-11-15T04:58:06.210 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.206+0000 7f9cbdffb700 1 -- 172.21.15.178:0/682607788 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cc0014dc0 con 0x7f9ccc060220 2023-11-15T04:58:06.211 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.206+0000 7f9cd3618700 1 -- 172.21.15.178:0/682607788 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ccc0f5680 con 0x7f9ccc060220 2023-11-15T04:58:06.211 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.207+0000 7f9cbdffb700 1 -- 172.21.15.178:0/682607788 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9cc0014420 con 0x7f9ccc060220 2023-11-15T04:58:06.211 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.207+0000 7f9cd3618700 1 -- 172.21.15.178:0/682607788 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ccc059100 con 0x7f9ccc060220 2023-11-15T04:58:06.211 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.207+0000 7f9cbdffb700 1 --2- 172.21.15.178:0/682607788 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f9cb802ffb0 0x7f9cb8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:06.211 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.207+0000 7f9cbdffb700 1 -- 172.21.15.178:0/682607788 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9cc0041100 con 0x7f9ccc060220 2023-11-15T04:58:06.212 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.207+0000 7f9cd0bb3700 1 --2- 172.21.15.178:0/682607788 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f9cb802ffb0 0x7f9cb8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:06.212 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.208+0000 7f9cd0bb3700 1 --2- 172.21.15.178:0/682607788 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f9cb802ffb0 0x7f9cb8032470 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f9cc80096a0 tx=0x7f9cc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:06.212 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.212+0000 7f9cbdffb700 1 -- 172.21.15.178:0/682607788 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9cc00095f0 con 0x7f9ccc060220 2023-11-15T04:58:06.419 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.419+0000 7f9cd3618700 1 -- 172.21.15.178:0/682607788 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ccc0fbf50 con 0x7f9ccc060220 2023-11-15T04:58:06.420 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.420+0000 7f9cbdffb700 1 -- 172.21.15.178:0/682607788 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9cc001db70 con 0x7f9ccc060220 2023-11-15T04:58:06.420 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:06.421 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:06.422 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.421+0000 7f9cd3618700 1 -- 172.21.15.178:0/682607788 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f9cb802ffb0 msgr2=0x7f9cb8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:06.422 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.421+0000 7f9cd3618700 1 --2- 172.21.15.178:0/682607788 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f9cb802ffb0 0x7f9cb8032470 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f9cc80096a0 tx=0x7f9cc8006b40).stop 2023-11-15T04:58:06.422 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.421+0000 7f9cd3618700 1 -- 172.21.15.178:0/682607788 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ccc060220 msgr2=0x7f9ccc0f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:06.422 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.421+0000 7f9cd3618700 1 --2- 172.21.15.178:0/682607788 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ccc060220 0x7f9ccc0f6bd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f9cc0013510 tx=0x7f9cc000b040).stop 2023-11-15T04:58:06.422 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.421+0000 7f9cd3618700 1 -- 172.21.15.178:0/682607788 shutdown_connections 2023-11-15T04:58:06.423 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.421+0000 7f9cd3618700 1 --2- 172.21.15.178:0/682607788 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f9cb802ffb0 0x7f9cb8032470 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:06.423 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.421+0000 7f9cd3618700 1 --2- 172.21.15.178:0/682607788 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ccc060220 0x7f9ccc0f6bd0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:06.423 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.421+0000 7f9cd3618700 1 -- 172.21.15.178:0/682607788 >> 172.21.15.178:0/682607788 conn(0x7f9ccc0f0b70 msgr2=0x7f9ccc0f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:06.423 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.421+0000 7f9cd3618700 1 -- 172.21.15.178:0/682607788 shutdown_connections 2023-11-15T04:58:06.423 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:06.421+0000 7f9cd3618700 1 -- 172.21.15.178:0/682607788 wait complete. 2023-11-15T04:58:06.424 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:58:07.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:06 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:06.420349+0000 mon.smithi163 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.178:0/682607788' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:58:07.852 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:58:07.853 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:08.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:07 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:06.333338+0000 mgr.smithi163.qesyrx (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:09.077 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.075+0000 7f7a4d480700 1 -- 172.21.15.178:0/2815760767 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a480f4b80 msgr2=0x7f7a480f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:09.077 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.075+0000 7f7a4d480700 1 --2- 172.21.15.178:0/2815760767 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a480f4b80 0x7f7a480f4fa0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f7a3c004770 tx=0x7f7a3c00ac40).stop 2023-11-15T04:58:09.078 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.075+0000 7f7a4d480700 1 -- 172.21.15.178:0/2815760767 shutdown_connections 2023-11-15T04:58:09.078 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.075+0000 7f7a4d480700 1 --2- 172.21.15.178:0/2815760767 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a480f4b80 0x7f7a480f4fa0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:09.078 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.075+0000 7f7a4d480700 1 -- 172.21.15.178:0/2815760767 >> 172.21.15.178:0/2815760767 conn(0x7f7a480f02e0 msgr2=0x7f7a480f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:09.078 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.075+0000 7f7a4d480700 1 -- 172.21.15.178:0/2815760767 shutdown_connections 2023-11-15T04:58:09.079 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.075+0000 7f7a4d480700 1 -- 172.21.15.178:0/2815760767 wait complete. 2023-11-15T04:58:09.079 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.076+0000 7f7a4d480700 1 Processor -- start 2023-11-15T04:58:09.079 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.076+0000 7f7a4d480700 1 -- start start 2023-11-15T04:58:09.079 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.076+0000 7f7a4d480700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a480f4b80 0x7f7a480fc280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:09.079 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.076+0000 7f7a4d480700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a480fa700 con 0x7f7a480f4b80 2023-11-15T04:58:09.080 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.076+0000 7f7a46ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a480f4b80 0x7f7a480fc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:09.080 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.076+0000 7f7a46ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a480f4b80 0x7f7a480fc280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:41508/0 (socket says 172.21.15.178:41508) 2023-11-15T04:58:09.080 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.076+0000 7f7a46ffd700 1 -- 172.21.15.178:0/84904964 learned_addr learned my addr 172.21.15.178:0/84904964 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:09.080 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.076+0000 7f7a46ffd700 1 -- 172.21.15.178:0/84904964 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3c005040 con 0x7f7a480f4b80 2023-11-15T04:58:09.080 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.077+0000 7f7a46ffd700 1 --2- 172.21.15.178:0/84904964 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a480f4b80 0x7f7a480fc280 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f7a3c004740 tx=0x7f7a3c00ac40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:09.081 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.077+0000 7f7a2f7fe700 1 -- 172.21.15.178:0/84904964 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a3c019070 con 0x7f7a480f4b80 2023-11-15T04:58:09.081 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.077+0000 7f7a2f7fe700 1 -- 172.21.15.178:0/84904964 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a3c007e70 con 0x7f7a480f4b80 2023-11-15T04:58:09.081 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.077+0000 7f7a4d480700 1 -- 172.21.15.178:0/84904964 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a480fa8a0 con 0x7f7a480f4b80 2023-11-15T04:58:09.081 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.077+0000 7f7a2f7fe700 1 -- 172.21.15.178:0/84904964 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a3c0082d0 con 0x7f7a480f4b80 2023-11-15T04:58:09.081 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.077+0000 7f7a4d480700 1 -- 172.21.15.178:0/84904964 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a480fad30 con 0x7f7a480f4b80 2023-11-15T04:58:09.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.077+0000 7f7a2f7fe700 1 -- 172.21.15.178:0/84904964 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a3c0084d0 con 0x7f7a480f4b80 2023-11-15T04:58:09.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.078+0000 7f7a4d480700 1 -- 172.21.15.178:0/84904964 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a48059100 con 0x7f7a480f4b80 2023-11-15T04:58:09.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.078+0000 7f7a2f7fe700 1 --2- 172.21.15.178:0/84904964 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7a3002ffb0 0x7f7a30032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:09.084 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.078+0000 7f7a2f7fe700 1 -- 172.21.15.178:0/84904964 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a3c040d50 con 0x7f7a480f4b80 2023-11-15T04:58:09.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.078+0000 7f7a467fc700 1 --2- 172.21.15.178:0/84904964 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7a3002ffb0 0x7f7a30032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:09.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.078+0000 7f7a467fc700 1 --2- 172.21.15.178:0/84904964 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7a3002ffb0 0x7f7a30032470 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f7a380096a0 tx=0x7f7a38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:09.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.083+0000 7f7a2f7fe700 1 -- 172.21.15.178:0/84904964 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a3c01d7f0 con 0x7f7a480f4b80 2023-11-15T04:58:09.287 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.287+0000 7f7a4d480700 1 -- 172.21.15.178:0/84904964 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a480f9d40 con 0x7f7a480f4b80 2023-11-15T04:58:09.288 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.287+0000 7f7a2f7fe700 1 -- 172.21.15.178:0/84904964 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7a3c01d7f0 con 0x7f7a480f4b80 2023-11-15T04:58:09.288 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:09.288 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:09.290 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.289+0000 7f7a4d480700 1 -- 172.21.15.178:0/84904964 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7a3002ffb0 msgr2=0x7f7a30032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:09.290 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.289+0000 7f7a4d480700 1 --2- 172.21.15.178:0/84904964 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7a3002ffb0 0x7f7a30032470 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f7a380096a0 tx=0x7f7a38006b40).stop 2023-11-15T04:58:09.290 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.289+0000 7f7a4d480700 1 -- 172.21.15.178:0/84904964 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a480f4b80 msgr2=0x7f7a480fc280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:09.291 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.289+0000 7f7a4d480700 1 --2- 172.21.15.178:0/84904964 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a480f4b80 0x7f7a480fc280 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f7a3c004740 tx=0x7f7a3c00ac40).stop 2023-11-15T04:58:09.291 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.289+0000 7f7a4d480700 1 -- 172.21.15.178:0/84904964 shutdown_connections 2023-11-15T04:58:09.291 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.289+0000 7f7a4d480700 1 --2- 172.21.15.178:0/84904964 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7a3002ffb0 0x7f7a30032470 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:09.291 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.289+0000 7f7a4d480700 1 --2- 172.21.15.178:0/84904964 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a480f4b80 0x7f7a480fc280 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:09.292 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.289+0000 7f7a4d480700 1 -- 172.21.15.178:0/84904964 >> 172.21.15.178:0/84904964 conn(0x7f7a480f02e0 msgr2=0x7f7a480f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:09.292 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.289+0000 7f7a4d480700 1 -- 172.21.15.178:0/84904964 shutdown_connections 2023-11-15T04:58:09.292 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:09.289+0000 7f7a4d480700 1 -- 172.21.15.178:0/84904964 wait complete. 2023-11-15T04:58:09.293 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:58:10.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:09 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:08.333665+0000 mgr.smithi163.qesyrx (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:10.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:09 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:09.288280+0000 mon.smithi163 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.178:0/84904964' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:58:10.695 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:58:10.696 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:11.909 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.909+0000 7f88e4f00700 1 -- 172.21.15.178:0/1470822712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88e00f6510 msgr2=0x7f88e00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:11.910 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.909+0000 7f88e4f00700 1 --2- 172.21.15.178:0/1470822712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88e00f6510 0x7f88e00f6930 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f88d4009fb0 tx=0x7f88d4009b30).stop 2023-11-15T04:58:11.910 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.909+0000 7f88e4f00700 1 -- 172.21.15.178:0/1470822712 shutdown_connections 2023-11-15T04:58:11.910 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.909+0000 7f88e4f00700 1 --2- 172.21.15.178:0/1470822712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88e00f6510 0x7f88e00f6930 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:11.910 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.909+0000 7f88e4f00700 1 -- 172.21.15.178:0/1470822712 >> 172.21.15.178:0/1470822712 conn(0x7f88e00f1c70 msgr2=0x7f88e00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:11.911 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.909+0000 7f88e4f00700 1 -- 172.21.15.178:0/1470822712 shutdown_connections 2023-11-15T04:58:11.911 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.909+0000 7f88e4f00700 1 -- 172.21.15.178:0/1470822712 wait complete. 2023-11-15T04:58:11.913 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.909+0000 7f88e4f00700 1 Processor -- start 2023-11-15T04:58:11.913 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.910+0000 7f88e4f00700 1 -- start start 2023-11-15T04:58:11.913 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.910+0000 7f88e4f00700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88e00f6510 0x7f88e0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:11.914 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.910+0000 7f88e4f00700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e0106c20 con 0x7f88e00f6510 2023-11-15T04:58:11.914 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.910+0000 7f88de59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88e00f6510 0x7f88e0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:11.914 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.910+0000 7f88de59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88e00f6510 0x7f88e0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:41530/0 (socket says 172.21.15.178:41530) 2023-11-15T04:58:11.914 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.910+0000 7f88de59c700 1 -- 172.21.15.178:0/1891662555 learned_addr learned my addr 172.21.15.178:0/1891662555 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:11.914 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.910+0000 7f88de59c700 1 -- 172.21.15.178:0/1891662555 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d4005040 con 0x7f88e00f6510 2023-11-15T04:58:11.915 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.910+0000 7f88de59c700 1 --2- 172.21.15.178:0/1891662555 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88e00f6510 0x7f88e0106730 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f88d4013510 tx=0x7f88d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:11.915 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.910+0000 7f88ceffd700 1 -- 172.21.15.178:0/1891662555 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88d4009320 con 0x7f88e00f6510 2023-11-15T04:58:11.915 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.911+0000 7f88ceffd700 1 -- 172.21.15.178:0/1891662555 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f88d4013f00 con 0x7f88e00f6510 2023-11-15T04:58:11.915 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.911+0000 7f88e4f00700 1 -- 172.21.15.178:0/1891662555 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e0106e20 con 0x7f88e00f6510 2023-11-15T04:58:11.916 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.911+0000 7f88ceffd700 1 -- 172.21.15.178:0/1891662555 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88d4014d90 con 0x7f88e00f6510 2023-11-15T04:58:11.916 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.911+0000 7f88e4f00700 1 -- 172.21.15.178:0/1891662555 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e01072b0 con 0x7f88e00f6510 2023-11-15T04:58:11.916 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.911+0000 7f88ceffd700 1 -- 172.21.15.178:0/1891662555 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f88d40263a0 con 0x7f88e00f6510 2023-11-15T04:58:11.916 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.911+0000 7f88ceffd700 1 --2- 172.21.15.178:0/1891662555 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f88c802ff60 0x7f88c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:11.916 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.912+0000 7f88ceffd700 1 -- 172.21.15.178:0/1891662555 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f88d4040ff0 con 0x7f88e00f6510 2023-11-15T04:58:11.917 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.912+0000 7f88e4f00700 1 -- 172.21.15.178:0/1891662555 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e0045ad0 con 0x7f88e00f6510 2023-11-15T04:58:11.917 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.912+0000 7f88ddd9b700 1 --2- 172.21.15.178:0/1891662555 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f88c802ff60 0x7f88c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:11.917 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.912+0000 7f88ddd9b700 1 --2- 172.21.15.178:0/1891662555 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f88c802ff60 0x7f88c8032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f88d00096a0 tx=0x7f88d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:11.918 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:11.916+0000 7f88ceffd700 1 -- 172.21.15.178:0/1891662555 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f88d4012920 con 0x7f88e00f6510 2023-11-15T04:58:12.121 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:12.121+0000 7f88e4f00700 1 -- 172.21.15.178:0/1891662555 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88e0107fd0 con 0x7f88e00f6510 2023-11-15T04:58:12.122 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:12.122+0000 7f88ceffd700 1 -- 172.21.15.178:0/1891662555 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f88d402b020 con 0x7f88e00f6510 2023-11-15T04:58:12.122 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:12.123 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:12.124 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:12.123+0000 7f88e4f00700 1 -- 172.21.15.178:0/1891662555 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f88c802ff60 msgr2=0x7f88c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:12.124 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:12.123+0000 7f88e4f00700 1 --2- 172.21.15.178:0/1891662555 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f88c802ff60 0x7f88c8032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f88d00096a0 tx=0x7f88d0006b40).stop 2023-11-15T04:58:12.124 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:12.123+0000 7f88e4f00700 1 -- 172.21.15.178:0/1891662555 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88e00f6510 msgr2=0x7f88e0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:12.124 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:12.123+0000 7f88e4f00700 1 --2- 172.21.15.178:0/1891662555 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88e00f6510 0x7f88e0106730 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f88d4013510 tx=0x7f88d400b040).stop 2023-11-15T04:58:12.124 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:12.123+0000 7f88e4f00700 1 -- 172.21.15.178:0/1891662555 shutdown_connections 2023-11-15T04:58:12.125 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:12.123+0000 7f88e4f00700 1 --2- 172.21.15.178:0/1891662555 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f88c802ff60 0x7f88c8032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:12.125 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:12.123+0000 7f88e4f00700 1 --2- 172.21.15.178:0/1891662555 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88e00f6510 0x7f88e0106730 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:12.125 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:12.123+0000 7f88e4f00700 1 -- 172.21.15.178:0/1891662555 >> 172.21.15.178:0/1891662555 conn(0x7f88e00f1c70 msgr2=0x7f88e00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:12.125 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:12.123+0000 7f88e4f00700 1 -- 172.21.15.178:0/1891662555 shutdown_connections 2023-11-15T04:58:12.125 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:12.123+0000 7f88e4f00700 1 -- 172.21.15.178:0/1891662555 wait complete. 2023-11-15T04:58:12.126 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:58:12.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:11 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:10.334028+0000 mgr.smithi163.qesyrx (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:13.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:12 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:12.122378+0000 mon.smithi163 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.178:0/1891662555' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:58:13.515 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:58:13.516 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:14.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:13 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:12.334338+0000 mgr.smithi163.qesyrx (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:14.735 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.733+0000 7ff7b628e700 1 -- 172.21.15.178:0/3266660053 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7b00f5410 msgr2=0x7ff7b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:14.736 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.733+0000 7ff7b628e700 1 --2- 172.21.15.178:0/3266660053 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7b00f5410 0x7ff7b00f5830 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7ff7a0009fb0 tx=0x7ff7a0009b30).stop 2023-11-15T04:58:14.736 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.733+0000 7ff7b628e700 1 -- 172.21.15.178:0/3266660053 shutdown_connections 2023-11-15T04:58:14.736 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.733+0000 7ff7b628e700 1 --2- 172.21.15.178:0/3266660053 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7b00f5410 0x7ff7b00f5830 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:14.736 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.733+0000 7ff7b628e700 1 -- 172.21.15.178:0/3266660053 >> 172.21.15.178:0/3266660053 conn(0x7ff7b00f0b70 msgr2=0x7ff7b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:14.736 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.733+0000 7ff7b628e700 1 -- 172.21.15.178:0/3266660053 shutdown_connections 2023-11-15T04:58:14.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.733+0000 7ff7b628e700 1 -- 172.21.15.178:0/3266660053 wait complete. 2023-11-15T04:58:14.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.733+0000 7ff7b628e700 1 Processor -- start 2023-11-15T04:58:14.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.734+0000 7ff7b628e700 1 -- start start 2023-11-15T04:58:14.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.734+0000 7ff7b628e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7b00f5410 0x7ff7b0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:14.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.734+0000 7ff7b628e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b0106c20 con 0x7ff7b00f5410 2023-11-15T04:58:14.738 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.734+0000 7ff7affff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7b00f5410 0x7ff7b0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:14.738 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.734+0000 7ff7affff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7b00f5410 0x7ff7b0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:51634/0 (socket says 172.21.15.178:51634) 2023-11-15T04:58:14.738 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.734+0000 7ff7affff700 1 -- 172.21.15.178:0/21457086 learned_addr learned my addr 172.21.15.178:0/21457086 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:14.738 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.734+0000 7ff7affff700 1 -- 172.21.15.178:0/21457086 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a0005040 con 0x7ff7b00f5410 2023-11-15T04:58:14.738 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.734+0000 7ff7affff700 1 --2- 172.21.15.178:0/21457086 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7b00f5410 0x7ff7b0106730 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7ff7a0013510 tx=0x7ff7a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:14.738 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.735+0000 7ff7acff9700 1 -- 172.21.15.178:0/21457086 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7a0009320 con 0x7ff7b00f5410 2023-11-15T04:58:14.739 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.735+0000 7ff7b628e700 1 -- 172.21.15.178:0/21457086 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b0106e20 con 0x7ff7b00f5410 2023-11-15T04:58:14.739 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.735+0000 7ff7acff9700 1 -- 172.21.15.178:0/21457086 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7a0013f00 con 0x7ff7b00f5410 2023-11-15T04:58:14.739 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.735+0000 7ff7acff9700 1 -- 172.21.15.178:0/21457086 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7a0014d90 con 0x7ff7b00f5410 2023-11-15T04:58:14.739 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.735+0000 7ff7b628e700 1 -- 172.21.15.178:0/21457086 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b01072b0 con 0x7ff7b00f5410 2023-11-15T04:58:14.740 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.735+0000 7ff7acff9700 1 -- 172.21.15.178:0/21457086 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7a00263a0 con 0x7ff7b00f5410 2023-11-15T04:58:14.741 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.736+0000 7ff7b628e700 1 -- 172.21.15.178:0/21457086 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7b0045ad0 con 0x7ff7b00f5410 2023-11-15T04:58:14.741 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.736+0000 7ff7acff9700 1 --2- 172.21.15.178:0/21457086 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff79802ff60 0x7ff798032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:14.741 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.736+0000 7ff7acff9700 1 -- 172.21.15.178:0/21457086 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff7a0041000 con 0x7ff7b00f5410 2023-11-15T04:58:14.741 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.736+0000 7ff7af7fe700 1 --2- 172.21.15.178:0/21457086 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff79802ff60 0x7ff798032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:14.741 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.736+0000 7ff7af7fe700 1 --2- 172.21.15.178:0/21457086 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff79802ff60 0x7ff798032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7ff7a40096a0 tx=0x7ff7a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:14.742 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.740+0000 7ff7acff9700 1 -- 172.21.15.178:0/21457086 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff7a00095f0 con 0x7ff7b00f5410 2023-11-15T04:58:14.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.946+0000 7ff7b628e700 1 -- 172.21.15.178:0/21457086 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7b00fbf90 con 0x7ff7b00f5410 2023-11-15T04:58:14.947 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.947+0000 7ff7acff9700 1 -- 172.21.15.178:0/21457086 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff7a001db70 con 0x7ff7b00f5410 2023-11-15T04:58:14.947 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:14.948 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:14.949 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.949+0000 7ff7b628e700 1 -- 172.21.15.178:0/21457086 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff79802ff60 msgr2=0x7ff798032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:14.949 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.949+0000 7ff7b628e700 1 --2- 172.21.15.178:0/21457086 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff79802ff60 0x7ff798032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7ff7a40096a0 tx=0x7ff7a4006b40).stop 2023-11-15T04:58:14.949 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.949+0000 7ff7b628e700 1 -- 172.21.15.178:0/21457086 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7b00f5410 msgr2=0x7ff7b0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:14.950 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.949+0000 7ff7b628e700 1 --2- 172.21.15.178:0/21457086 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7b00f5410 0x7ff7b0106730 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7ff7a0013510 tx=0x7ff7a000b040).stop 2023-11-15T04:58:14.950 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.949+0000 7ff7b628e700 1 -- 172.21.15.178:0/21457086 shutdown_connections 2023-11-15T04:58:14.950 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.949+0000 7ff7b628e700 1 --2- 172.21.15.178:0/21457086 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff79802ff60 0x7ff798032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:14.950 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.949+0000 7ff7b628e700 1 --2- 172.21.15.178:0/21457086 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7b00f5410 0x7ff7b0106730 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:14.951 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.949+0000 7ff7b628e700 1 -- 172.21.15.178:0/21457086 >> 172.21.15.178:0/21457086 conn(0x7ff7b00f0b70 msgr2=0x7ff7b00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:14.951 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.949+0000 7ff7b628e700 1 -- 172.21.15.178:0/21457086 shutdown_connections 2023-11-15T04:58:14.951 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:14.949+0000 7ff7b628e700 1 -- 172.21.15.178:0/21457086 wait complete. 2023-11-15T04:58:14.951 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:58:16.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:15 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:14.334712+0000 mgr.smithi163.qesyrx (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:16.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:15 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:14.947473+0000 mon.smithi163 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.178:0/21457086' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:58:16.331 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:58:16.332 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:17.556 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.554+0000 7fd7bb4ec700 1 -- 172.21.15.178:0/2638777642 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7b40f5410 msgr2=0x7fd7b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:17.556 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.554+0000 7fd7bb4ec700 1 --2- 172.21.15.178:0/2638777642 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7b40f5410 0x7fd7b40f5830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fd7b0009fb0 tx=0x7fd7b0009b30).stop 2023-11-15T04:58:17.556 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.554+0000 7fd7bb4ec700 1 -- 172.21.15.178:0/2638777642 shutdown_connections 2023-11-15T04:58:17.556 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.554+0000 7fd7bb4ec700 1 --2- 172.21.15.178:0/2638777642 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7b40f5410 0x7fd7b40f5830 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:17.557 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.554+0000 7fd7bb4ec700 1 -- 172.21.15.178:0/2638777642 >> 172.21.15.178:0/2638777642 conn(0x7fd7b40f0b70 msgr2=0x7fd7b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:17.557 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.554+0000 7fd7bb4ec700 1 -- 172.21.15.178:0/2638777642 shutdown_connections 2023-11-15T04:58:17.557 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.554+0000 7fd7bb4ec700 1 -- 172.21.15.178:0/2638777642 wait complete. 2023-11-15T04:58:17.557 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.554+0000 7fd7bb4ec700 1 Processor -- start 2023-11-15T04:58:17.557 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.555+0000 7fd7bb4ec700 1 -- start start 2023-11-15T04:58:17.558 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.555+0000 7fd7bb4ec700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7b40f5410 0x7fd7b4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:17.558 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.555+0000 7fd7bb4ec700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b4060240 con 0x7fd7b40f5410 2023-11-15T04:58:17.560 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.555+0000 7fd7b9288700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7b40f5410 0x7fd7b4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:17.561 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.555+0000 7fd7b9288700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7b40f5410 0x7fd7b4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:51650/0 (socket says 172.21.15.178:51650) 2023-11-15T04:58:17.561 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.555+0000 7fd7b9288700 1 -- 172.21.15.178:0/3463673095 learned_addr learned my addr 172.21.15.178:0/3463673095 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:17.561 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.555+0000 7fd7b9288700 1 -- 172.21.15.178:0/3463673095 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b0005040 con 0x7fd7b40f5410 2023-11-15T04:58:17.561 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.555+0000 7fd7b9288700 1 --2- 172.21.15.178:0/3463673095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7b40f5410 0x7fd7b4109430 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fd7b0013510 tx=0x7fd7b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:17.561 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.556+0000 7fd7a9ffb700 1 -- 172.21.15.178:0/3463673095 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7b0009320 con 0x7fd7b40f5410 2023-11-15T04:58:17.562 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.556+0000 7fd7bb4ec700 1 -- 172.21.15.178:0/3463673095 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7b4060440 con 0x7fd7b40f5410 2023-11-15T04:58:17.562 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.556+0000 7fd7a9ffb700 1 -- 172.21.15.178:0/3463673095 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7b0013f00 con 0x7fd7b40f5410 2023-11-15T04:58:17.562 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.556+0000 7fd7a9ffb700 1 -- 172.21.15.178:0/3463673095 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7b0014d90 con 0x7fd7b40f5410 2023-11-15T04:58:17.562 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.556+0000 7fd7bb4ec700 1 -- 172.21.15.178:0/3463673095 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7b4109ae0 con 0x7fd7b40f5410 2023-11-15T04:58:17.562 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.557+0000 7fd7a9ffb700 1 -- 172.21.15.178:0/3463673095 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd7b00095f0 con 0x7fd7b40f5410 2023-11-15T04:58:17.563 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.557+0000 7fd7bb4ec700 1 -- 172.21.15.178:0/3463673095 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7b4045ad0 con 0x7fd7b40f5410 2023-11-15T04:58:17.563 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.557+0000 7fd7a9ffb700 1 --2- 172.21.15.178:0/3463673095 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd7a002ff60 0x7fd7a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:17.563 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.557+0000 7fd7a9ffb700 1 -- 172.21.15.178:0/3463673095 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd7b0041000 con 0x7fd7b40f5410 2023-11-15T04:58:17.563 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.557+0000 7fd7b8a87700 1 --2- 172.21.15.178:0/3463673095 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd7a002ff60 0x7fd7a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:17.563 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.557+0000 7fd7b8a87700 1 --2- 172.21.15.178:0/3463673095 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd7a002ff60 0x7fd7a0032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fd7a40096a0 tx=0x7fd7a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:17.564 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.562+0000 7fd7a9ffb700 1 -- 172.21.15.178:0/3463673095 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd7b001d7f0 con 0x7fd7b40f5410 2023-11-15T04:58:17.767 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.766+0000 7fd7bb4ec700 1 -- 172.21.15.178:0/3463673095 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd7b40fae60 con 0x7fd7b40f5410 2023-11-15T04:58:17.768 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.766+0000 7fd7a9ffb700 1 -- 172.21.15.178:0/3463673095 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd7b0010230 con 0x7fd7b40f5410 2023-11-15T04:58:17.768 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:17.769 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:17.769 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.768+0000 7fd7bb4ec700 1 -- 172.21.15.178:0/3463673095 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd7a002ff60 msgr2=0x7fd7a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:17.770 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.768+0000 7fd7bb4ec700 1 --2- 172.21.15.178:0/3463673095 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd7a002ff60 0x7fd7a0032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fd7a40096a0 tx=0x7fd7a4006b40).stop 2023-11-15T04:58:17.770 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.768+0000 7fd7bb4ec700 1 -- 172.21.15.178:0/3463673095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7b40f5410 msgr2=0x7fd7b4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:17.770 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.768+0000 7fd7bb4ec700 1 --2- 172.21.15.178:0/3463673095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7b40f5410 0x7fd7b4109430 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fd7b0013510 tx=0x7fd7b000b040).stop 2023-11-15T04:58:17.770 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.768+0000 7fd7bb4ec700 1 -- 172.21.15.178:0/3463673095 shutdown_connections 2023-11-15T04:58:17.770 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.768+0000 7fd7bb4ec700 1 --2- 172.21.15.178:0/3463673095 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd7a002ff60 0x7fd7a0032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:17.771 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.768+0000 7fd7bb4ec700 1 --2- 172.21.15.178:0/3463673095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7b40f5410 0x7fd7b4109430 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:17.771 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.768+0000 7fd7bb4ec700 1 -- 172.21.15.178:0/3463673095 >> 172.21.15.178:0/3463673095 conn(0x7fd7b40f0b70 msgr2=0x7fd7b40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:17.771 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.768+0000 7fd7bb4ec700 1 -- 172.21.15.178:0/3463673095 shutdown_connections 2023-11-15T04:58:17.771 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:17.768+0000 7fd7bb4ec700 1 -- 172.21.15.178:0/3463673095 wait complete. 2023-11-15T04:58:17.772 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:58:18.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:17 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:16.335083+0000 mgr.smithi163.qesyrx (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:18.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:17 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:17.768182+0000 mon.smithi163 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.178:0/3463673095' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:58:19.164 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:58:19.164 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:20.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:19 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:18.335375+0000 mgr.smithi163.qesyrx (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:20.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.338+0000 7f8f08618700 1 -- 172.21.15.178:0/2411542386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f000f6990 msgr2=0x7f8f000f6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:20.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.338+0000 7f8f08618700 1 --2- 172.21.15.178:0/2411542386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f000f6990 0x7f8f000f6db0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f8ef0009fb0 tx=0x7f8ef0009b30).stop 2023-11-15T04:58:20.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.338+0000 7f8f08618700 1 -- 172.21.15.178:0/2411542386 shutdown_connections 2023-11-15T04:58:20.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.338+0000 7f8f08618700 1 --2- 172.21.15.178:0/2411542386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f000f6990 0x7f8f000f6db0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:20.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.338+0000 7f8f08618700 1 -- 172.21.15.178:0/2411542386 >> 172.21.15.178:0/2411542386 conn(0x7f8f000f0260 msgr2=0x7f8f000f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:20.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.338+0000 7f8f08618700 1 -- 172.21.15.178:0/2411542386 shutdown_connections 2023-11-15T04:58:20.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.338+0000 7f8f08618700 1 -- 172.21.15.178:0/2411542386 wait complete. 2023-11-15T04:58:20.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.339+0000 7f8f08618700 1 Processor -- start 2023-11-15T04:58:20.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.339+0000 7f8f08618700 1 -- start start 2023-11-15T04:58:20.343 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.339+0000 7f8f08618700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f000f6990 0x7f8f00106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:20.343 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.339+0000 7f8f08618700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f00106c10 con 0x7f8f000f6990 2023-11-15T04:58:20.343 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.339+0000 7f8f063b4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f000f6990 0x7f8f00106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:20.343 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.339+0000 7f8f063b4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f000f6990 0x7f8f00106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:51660/0 (socket says 172.21.15.178:51660) 2023-11-15T04:58:20.344 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.339+0000 7f8f063b4700 1 -- 172.21.15.178:0/4080353086 learned_addr learned my addr 172.21.15.178:0/4080353086 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:20.344 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.339+0000 7f8f063b4700 1 -- 172.21.15.178:0/4080353086 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef0005040 con 0x7f8f000f6990 2023-11-15T04:58:20.344 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.339+0000 7f8f063b4700 1 --2- 172.21.15.178:0/4080353086 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f000f6990 0x7f8f00106720 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f8ef0013510 tx=0x7f8ef000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:20.344 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.340+0000 7f8ef6ffd700 1 -- 172.21.15.178:0/4080353086 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ef0009320 con 0x7f8f000f6990 2023-11-15T04:58:20.344 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.340+0000 7f8ef6ffd700 1 -- 172.21.15.178:0/4080353086 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ef0013f00 con 0x7f8f000f6990 2023-11-15T04:58:20.344 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.340+0000 7f8f08618700 1 -- 172.21.15.178:0/4080353086 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f00106e10 con 0x7f8f000f6990 2023-11-15T04:58:20.345 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.340+0000 7f8ef6ffd700 1 -- 172.21.15.178:0/4080353086 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ef0014d90 con 0x7f8f000f6990 2023-11-15T04:58:20.345 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.340+0000 7f8f08618700 1 -- 172.21.15.178:0/4080353086 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f001072a0 con 0x7f8f000f6990 2023-11-15T04:58:20.346 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.340+0000 7f8ef6ffd700 1 -- 172.21.15.178:0/4080353086 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ef00263a0 con 0x7f8f000f6990 2023-11-15T04:58:20.346 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.341+0000 7f8ef6ffd700 1 --2- 172.21.15.178:0/4080353086 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8eec02ff60 0x7f8eec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:20.346 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.341+0000 7f8f08618700 1 -- 172.21.15.178:0/4080353086 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f00045ad0 con 0x7f8f000f6990 2023-11-15T04:58:20.346 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.341+0000 7f8ef6ffd700 1 -- 172.21.15.178:0/4080353086 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ef0041030 con 0x7f8f000f6990 2023-11-15T04:58:20.347 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.341+0000 7f8f05bb3700 1 --2- 172.21.15.178:0/4080353086 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8eec02ff60 0x7f8eec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:20.347 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.341+0000 7f8f05bb3700 1 --2- 172.21.15.178:0/4080353086 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8eec02ff60 0x7f8eec032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f8efc0096a0 tx=0x7f8efc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:20.347 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.346+0000 7f8ef6ffd700 1 -- 172.21.15.178:0/4080353086 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ef006b2b0 con 0x7f8f000f6990 2023-11-15T04:58:20.552 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.551+0000 7f8f08618700 1 -- 172.21.15.178:0/4080353086 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f001080f0 con 0x7f8f000f6990 2023-11-15T04:58:20.553 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.552+0000 7f8ef6ffd700 1 -- 172.21.15.178:0/4080353086 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8ef00126f0 con 0x7f8f000f6990 2023-11-15T04:58:20.553 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:20.554 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:20.555 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.553+0000 7f8f08618700 1 -- 172.21.15.178:0/4080353086 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8eec02ff60 msgr2=0x7f8eec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:20.555 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.553+0000 7f8f08618700 1 --2- 172.21.15.178:0/4080353086 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8eec02ff60 0x7f8eec032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f8efc0096a0 tx=0x7f8efc006b40).stop 2023-11-15T04:58:20.555 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.554+0000 7f8f08618700 1 -- 172.21.15.178:0/4080353086 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f000f6990 msgr2=0x7f8f00106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:20.556 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.554+0000 7f8f08618700 1 --2- 172.21.15.178:0/4080353086 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f000f6990 0x7f8f00106720 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f8ef0013510 tx=0x7f8ef000b040).stop 2023-11-15T04:58:20.556 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.554+0000 7f8f08618700 1 -- 172.21.15.178:0/4080353086 shutdown_connections 2023-11-15T04:58:20.556 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.554+0000 7f8f08618700 1 --2- 172.21.15.178:0/4080353086 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8eec02ff60 0x7f8eec032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:20.556 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.554+0000 7f8f08618700 1 --2- 172.21.15.178:0/4080353086 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f000f6990 0x7f8f00106720 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:20.556 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.554+0000 7f8f08618700 1 -- 172.21.15.178:0/4080353086 >> 172.21.15.178:0/4080353086 conn(0x7f8f000f0260 msgr2=0x7f8f000f4ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:20.557 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.554+0000 7f8f08618700 1 -- 172.21.15.178:0/4080353086 shutdown_connections 2023-11-15T04:58:20.557 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:20.554+0000 7f8f08618700 1 -- 172.21.15.178:0/4080353086 wait complete. 2023-11-15T04:58:20.557 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:58:21.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:20 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:20.553527+0000 mon.smithi163 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.178:0/4080353086' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:58:21.916 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:58:21.917 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:22.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:21 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:20.335622+0000 mgr.smithi163.qesyrx (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:23.115 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.113+0000 7f82a6bde700 1 -- 172.21.15.178:0/226274189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f82a00601e0 msgr2=0x7f82a00f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:23.115 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.113+0000 7f82a6bde700 1 --2- 172.21.15.178:0/226274189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f82a00601e0 0x7f82a00f9190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f8294009fb0 tx=0x7f8294009b30).stop 2023-11-15T04:58:23.116 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.113+0000 7f82a6bde700 1 -- 172.21.15.178:0/226274189 shutdown_connections 2023-11-15T04:58:23.116 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.113+0000 7f82a6bde700 1 --2- 172.21.15.178:0/226274189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f82a00601e0 0x7f82a00f9190 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:23.116 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.113+0000 7f82a6bde700 1 -- 172.21.15.178:0/226274189 >> 172.21.15.178:0/226274189 conn(0x7f82a00f0b70 msgr2=0x7f82a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:23.116 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.113+0000 7f82a6bde700 1 -- 172.21.15.178:0/226274189 shutdown_connections 2023-11-15T04:58:23.116 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.113+0000 7f82a6bde700 1 -- 172.21.15.178:0/226274189 wait complete. 2023-11-15T04:58:23.117 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.114+0000 7f82a6bde700 1 Processor -- start 2023-11-15T04:58:23.117 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.114+0000 7f82a6bde700 1 -- start start 2023-11-15T04:58:23.118 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.114+0000 7f82a6bde700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f82a00601e0 0x7f82a01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:23.119 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.114+0000 7f82a6bde700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a0106bd0 con 0x7f82a00601e0 2023-11-15T04:58:23.119 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.114+0000 7f82a497a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f82a00601e0 0x7f82a01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:23.119 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.114+0000 7f82a497a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f82a00601e0 0x7f82a01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:51692/0 (socket says 172.21.15.178:51692) 2023-11-15T04:58:23.119 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.114+0000 7f82a497a700 1 -- 172.21.15.178:0/4236027888 learned_addr learned my addr 172.21.15.178:0/4236027888 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:23.119 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.115+0000 7f82a497a700 1 -- 172.21.15.178:0/4236027888 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8294005040 con 0x7f82a00601e0 2023-11-15T04:58:23.120 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.115+0000 7f82a497a700 1 --2- 172.21.15.178:0/4236027888 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f82a00601e0 0x7f82a01066e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f8294004dd0 tx=0x7f8294009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:23.120 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.115+0000 7f829d7fa700 1 -- 172.21.15.178:0/4236027888 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8294018070 con 0x7f82a00601e0 2023-11-15T04:58:23.120 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.115+0000 7f829d7fa700 1 -- 172.21.15.178:0/4236027888 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8294013410 con 0x7f82a00601e0 2023-11-15T04:58:23.120 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.115+0000 7f82a6bde700 1 -- 172.21.15.178:0/4236027888 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82a0106dd0 con 0x7f82a00601e0 2023-11-15T04:58:23.120 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.115+0000 7f829d7fa700 1 -- 172.21.15.178:0/4236027888 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f829401c3c0 con 0x7f82a00601e0 2023-11-15T04:58:23.121 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.115+0000 7f82a6bde700 1 -- 172.21.15.178:0/4236027888 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a0107260 con 0x7f82a00601e0 2023-11-15T04:58:23.121 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.116+0000 7f829d7fa700 1 -- 172.21.15.178:0/4236027888 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f829401d420 con 0x7f82a00601e0 2023-11-15T04:58:23.122 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.116+0000 7f82a6bde700 1 -- 172.21.15.178:0/4236027888 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82a0045ad0 con 0x7f82a00601e0 2023-11-15T04:58:23.122 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.116+0000 7f829d7fa700 1 --2- 172.21.15.178:0/4236027888 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f829002ff60 0x7f8290032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:23.122 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.116+0000 7f829d7fa700 1 -- 172.21.15.178:0/4236027888 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8294041310 con 0x7f82a00601e0 2023-11-15T04:58:23.122 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.116+0000 7f829ffff700 1 --2- 172.21.15.178:0/4236027888 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f829002ff60 0x7f8290032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:23.122 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.117+0000 7f829ffff700 1 --2- 172.21.15.178:0/4236027888 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f829002ff60 0x7f8290032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f828c0096a0 tx=0x7f828c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:23.123 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.121+0000 7f829d7fa700 1 -- 172.21.15.178:0/4236027888 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f829401d820 con 0x7f82a00601e0 2023-11-15T04:58:23.325 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.324+0000 7f82a6bde700 1 -- 172.21.15.178:0/4236027888 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f82a0108110 con 0x7f82a00601e0 2023-11-15T04:58:23.325 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.324+0000 7f829d7fa700 1 -- 172.21.15.178:0/4236027888 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f829401b370 con 0x7f82a00601e0 2023-11-15T04:58:23.326 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:23.326 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:23.327 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.326+0000 7f82a6bde700 1 -- 172.21.15.178:0/4236027888 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f829002ff60 msgr2=0x7f8290032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:23.327 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.326+0000 7f82a6bde700 1 --2- 172.21.15.178:0/4236027888 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f829002ff60 0x7f8290032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f828c0096a0 tx=0x7f828c006b40).stop 2023-11-15T04:58:23.328 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.326+0000 7f82a6bde700 1 -- 172.21.15.178:0/4236027888 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f82a00601e0 msgr2=0x7f82a01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:23.328 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.326+0000 7f82a6bde700 1 --2- 172.21.15.178:0/4236027888 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f82a00601e0 0x7f82a01066e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f8294004dd0 tx=0x7f8294009b30).stop 2023-11-15T04:58:23.328 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.326+0000 7f82a6bde700 1 -- 172.21.15.178:0/4236027888 shutdown_connections 2023-11-15T04:58:23.328 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.326+0000 7f82a6bde700 1 --2- 172.21.15.178:0/4236027888 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f829002ff60 0x7f8290032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:23.328 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.326+0000 7f82a6bde700 1 --2- 172.21.15.178:0/4236027888 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f82a00601e0 0x7f82a01066e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:23.329 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.326+0000 7f82a6bde700 1 -- 172.21.15.178:0/4236027888 >> 172.21.15.178:0/4236027888 conn(0x7f82a00f0b70 msgr2=0x7f82a00f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:23.329 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.326+0000 7f82a6bde700 1 -- 172.21.15.178:0/4236027888 shutdown_connections 2023-11-15T04:58:23.329 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:23.326+0000 7f82a6bde700 1 -- 172.21.15.178:0/4236027888 wait complete. 2023-11-15T04:58:23.329 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:58:24.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:23 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:22.335867+0000 mgr.smithi163.qesyrx (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:24.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:23 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:23.326031+0000 mon.smithi163 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.178:0/4236027888' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:58:24.729 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:58:24.729 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:25.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:24 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:24.450484+0000 mon.smithi163 (mon.0) 174 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:58:25.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:24 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:24.452209+0000 mon.smithi163 (mon.0) 175 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/trash_purge_schedule"}]: dispatch 2023-11-15T04:58:25.937 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.934+0000 7f68aa726700 1 -- 172.21.15.178:0/3896414630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68a40f62a0 msgr2=0x7f68a40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:25.938 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.934+0000 7f68aa726700 1 --2- 172.21.15.178:0/3896414630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68a40f62a0 0x7f68a40f66c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f68a0009fb0 tx=0x7f68a0009b30).stop 2023-11-15T04:58:25.938 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.934+0000 7f68aa726700 1 -- 172.21.15.178:0/3896414630 shutdown_connections 2023-11-15T04:58:25.938 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.934+0000 7f68aa726700 1 --2- 172.21.15.178:0/3896414630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68a40f62a0 0x7f68a40f66c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:25.939 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.934+0000 7f68aa726700 1 -- 172.21.15.178:0/3896414630 >> 172.21.15.178:0/3896414630 conn(0x7f68a40f1aa0 msgr2=0x7f68a40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:25.939 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.934+0000 7f68aa726700 1 -- 172.21.15.178:0/3896414630 shutdown_connections 2023-11-15T04:58:25.939 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.934+0000 7f68aa726700 1 -- 172.21.15.178:0/3896414630 wait complete. 2023-11-15T04:58:25.940 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.935+0000 7f68aa726700 1 Processor -- start 2023-11-15T04:58:25.940 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.935+0000 7f68aa726700 1 -- start start 2023-11-15T04:58:25.940 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.935+0000 7f68aa726700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68a40f62a0 0x7f68a41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:25.940 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.935+0000 7f68aa726700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a41069a0 con 0x7f68a40f62a0 2023-11-15T04:58:25.941 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.935+0000 7f68a9724700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68a40f62a0 0x7f68a41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:25.941 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.935+0000 7f68a9724700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68a40f62a0 0x7f68a41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:39776/0 (socket says 172.21.15.178:39776) 2023-11-15T04:58:25.941 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.935+0000 7f68a9724700 1 -- 172.21.15.178:0/3376942409 learned_addr learned my addr 172.21.15.178:0/3376942409 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:25.941 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.935+0000 7f68a9724700 1 -- 172.21.15.178:0/3376942409 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a0005040 con 0x7f68a40f62a0 2023-11-15T04:58:25.942 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.935+0000 7f68a9724700 1 --2- 172.21.15.178:0/3376942409 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68a40f62a0 0x7f68a41064b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f68a0013510 tx=0x7f68a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:25.942 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.936+0000 7f6899ffb700 1 -- 172.21.15.178:0/3376942409 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68a0009320 con 0x7f68a40f62a0 2023-11-15T04:58:25.942 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.936+0000 7f68aa726700 1 -- 172.21.15.178:0/3376942409 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a4106ba0 con 0x7f68a40f62a0 2023-11-15T04:58:25.942 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.936+0000 7f6899ffb700 1 -- 172.21.15.178:0/3376942409 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68a0013f00 con 0x7f68a40f62a0 2023-11-15T04:58:25.943 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.936+0000 7f6899ffb700 1 -- 172.21.15.178:0/3376942409 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68a0014dc0 con 0x7f68a40f62a0 2023-11-15T04:58:25.943 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.936+0000 7f68aa726700 1 -- 172.21.15.178:0/3376942409 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a4107030 con 0x7f68a40f62a0 2023-11-15T04:58:25.945 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.937+0000 7f6899ffb700 1 -- 172.21.15.178:0/3376942409 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68a00095f0 con 0x7f68a40f62a0 2023-11-15T04:58:25.945 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.937+0000 7f68aa726700 1 -- 172.21.15.178:0/3376942409 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a4045ad0 con 0x7f68a40f62a0 2023-11-15T04:58:25.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.937+0000 7f6899ffb700 1 --2- 172.21.15.178:0/3376942409 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f689002ff60 0x7f6890032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:25.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.937+0000 7f6899ffb700 1 -- 172.21.15.178:0/3376942409 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f68a0041040 con 0x7f68a40f62a0 2023-11-15T04:58:25.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.937+0000 7f68a8f23700 1 --2- 172.21.15.178:0/3376942409 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f689002ff60 0x7f6890032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:25.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.938+0000 7f68a8f23700 1 --2- 172.21.15.178:0/3376942409 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f689002ff60 0x7f6890032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f68940096a0 tx=0x7f6894006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:25.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:25.942+0000 7f6899ffb700 1 -- 172.21.15.178:0/3376942409 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f68a0014420 con 0x7f68a40f62a0 2023-11-15T04:58:26.146 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:26.145+0000 7f68aa726700 1 -- 172.21.15.178:0/3376942409 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f68a4107ee0 con 0x7f68a40f62a0 2023-11-15T04:58:26.147 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:26.145+0000 7f6899ffb700 1 -- 172.21.15.178:0/3376942409 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f68a0010230 con 0x7f68a40f62a0 2023-11-15T04:58:26.148 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:26.148 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:26.149 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:26.147+0000 7f68aa726700 1 -- 172.21.15.178:0/3376942409 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f689002ff60 msgr2=0x7f6890032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:26.150 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:26.147+0000 7f68aa726700 1 --2- 172.21.15.178:0/3376942409 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f689002ff60 0x7f6890032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f68940096a0 tx=0x7f6894006b40).stop 2023-11-15T04:58:26.150 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:26.147+0000 7f68aa726700 1 -- 172.21.15.178:0/3376942409 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68a40f62a0 msgr2=0x7f68a41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:26.150 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:26.147+0000 7f68aa726700 1 --2- 172.21.15.178:0/3376942409 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68a40f62a0 0x7f68a41064b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f68a0013510 tx=0x7f68a000b040).stop 2023-11-15T04:58:26.150 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:26.147+0000 7f68aa726700 1 -- 172.21.15.178:0/3376942409 shutdown_connections 2023-11-15T04:58:26.150 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:26.147+0000 7f68aa726700 1 --2- 172.21.15.178:0/3376942409 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f689002ff60 0x7f6890032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:26.151 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:26.147+0000 7f68aa726700 1 --2- 172.21.15.178:0/3376942409 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68a40f62a0 0x7f68a41064b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:26.151 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:26.147+0000 7f68aa726700 1 -- 172.21.15.178:0/3376942409 >> 172.21.15.178:0/3376942409 conn(0x7f68a40f1aa0 msgr2=0x7f68a40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:26.151 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:26.147+0000 7f68aa726700 1 -- 172.21.15.178:0/3376942409 shutdown_connections 2023-11-15T04:58:26.151 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:26.147+0000 7f68aa726700 1 -- 172.21.15.178:0/3376942409 wait complete. 2023-11-15T04:58:26.151 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:58:26.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:24.336139+0000 mgr.smithi163.qesyrx (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:27.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:26 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:26.147208+0000 mon.smithi163 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.178:0/3376942409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:58:27.545 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:58:27.546 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:28.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:27 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:26.336459+0000 mgr.smithi163.qesyrx (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:28.682 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.680+0000 7f430015b700 1 -- 172.21.15.178:0/1153199727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42f80f5410 msgr2=0x7f42f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:28.682 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.680+0000 7f430015b700 1 --2- 172.21.15.178:0/1153199727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42f80f5410 0x7f42f80f5830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f42e8009fb0 tx=0x7f42e8009b30).stop 2023-11-15T04:58:28.683 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.680+0000 7f430015b700 1 -- 172.21.15.178:0/1153199727 shutdown_connections 2023-11-15T04:58:28.683 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.680+0000 7f430015b700 1 --2- 172.21.15.178:0/1153199727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42f80f5410 0x7f42f80f5830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:28.683 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.680+0000 7f430015b700 1 -- 172.21.15.178:0/1153199727 >> 172.21.15.178:0/1153199727 conn(0x7f42f80f0b70 msgr2=0x7f42f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:28.683 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.680+0000 7f430015b700 1 -- 172.21.15.178:0/1153199727 shutdown_connections 2023-11-15T04:58:28.683 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.680+0000 7f430015b700 1 -- 172.21.15.178:0/1153199727 wait complete. 2023-11-15T04:58:28.684 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.680+0000 7f430015b700 1 Processor -- start 2023-11-15T04:58:28.684 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.680+0000 7f430015b700 1 -- start start 2023-11-15T04:58:28.684 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.681+0000 7f430015b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42f80f5410 0x7f42f8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:28.684 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.681+0000 7f430015b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f8060240 con 0x7f42f80f5410 2023-11-15T04:58:28.684 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.681+0000 7f42fdef7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42f80f5410 0x7f42f8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:28.685 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.681+0000 7f42fdef7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42f80f5410 0x7f42f8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:39804/0 (socket says 172.21.15.178:39804) 2023-11-15T04:58:28.685 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.681+0000 7f42fdef7700 1 -- 172.21.15.178:0/3643388220 learned_addr learned my addr 172.21.15.178:0/3643388220 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:28.686 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.681+0000 7f42fdef7700 1 -- 172.21.15.178:0/3643388220 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e8005040 con 0x7f42f80f5410 2023-11-15T04:58:28.687 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.681+0000 7f42fdef7700 1 --2- 172.21.15.178:0/3643388220 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42f80f5410 0x7f42f8109430 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f42e8013510 tx=0x7f42e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:28.687 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.682+0000 7f42ee7fc700 1 -- 172.21.15.178:0/3643388220 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42e8009320 con 0x7f42f80f5410 2023-11-15T04:58:28.687 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.682+0000 7f42ee7fc700 1 -- 172.21.15.178:0/3643388220 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42e8013f00 con 0x7f42f80f5410 2023-11-15T04:58:28.687 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.682+0000 7f430015b700 1 -- 172.21.15.178:0/3643388220 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f8060440 con 0x7f42f80f5410 2023-11-15T04:58:28.687 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.682+0000 7f42ee7fc700 1 -- 172.21.15.178:0/3643388220 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42e8014d90 con 0x7f42f80f5410 2023-11-15T04:58:28.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.682+0000 7f430015b700 1 -- 172.21.15.178:0/3643388220 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f8109ae0 con 0x7f42f80f5410 2023-11-15T04:58:28.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.682+0000 7f42ee7fc700 1 -- 172.21.15.178:0/3643388220 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42e80095f0 con 0x7f42f80f5410 2023-11-15T04:58:28.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.682+0000 7f430015b700 1 -- 172.21.15.178:0/3643388220 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f8045ad0 con 0x7f42f80f5410 2023-11-15T04:58:28.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.683+0000 7f42ee7fc700 1 --2- 172.21.15.178:0/3643388220 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f42e402ff60 0x7f42e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:28.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.683+0000 7f42ee7fc700 1 -- 172.21.15.178:0/3643388220 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f42e8041000 con 0x7f42f80f5410 2023-11-15T04:58:28.689 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.683+0000 7f42fd6f6700 1 --2- 172.21.15.178:0/3643388220 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f42e402ff60 0x7f42e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:28.689 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.683+0000 7f42fd6f6700 1 --2- 172.21.15.178:0/3643388220 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f42e402ff60 0x7f42e4032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f42f40096a0 tx=0x7f42f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:28.689 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.687+0000 7f42ee7fc700 1 -- 172.21.15.178:0/3643388220 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42e8014420 con 0x7f42f80f5410 2023-11-15T04:58:28.897 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.895+0000 7f430015b700 1 -- 172.21.15.178:0/3643388220 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42f80fae60 con 0x7f42f80f5410 2023-11-15T04:58:28.898 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.897+0000 7f42ee7fc700 1 -- 172.21.15.178:0/3643388220 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f42e8010230 con 0x7f42f80f5410 2023-11-15T04:58:28.898 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:28.898 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:28.899 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.898+0000 7f430015b700 1 -- 172.21.15.178:0/3643388220 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f42e402ff60 msgr2=0x7f42e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:28.899 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.898+0000 7f430015b700 1 --2- 172.21.15.178:0/3643388220 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f42e402ff60 0x7f42e4032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f42f40096a0 tx=0x7f42f4006b40).stop 2023-11-15T04:58:28.900 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.898+0000 7f430015b700 1 -- 172.21.15.178:0/3643388220 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42f80f5410 msgr2=0x7f42f8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:28.900 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.898+0000 7f430015b700 1 --2- 172.21.15.178:0/3643388220 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42f80f5410 0x7f42f8109430 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f42e8013510 tx=0x7f42e800b040).stop 2023-11-15T04:58:28.900 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.898+0000 7f430015b700 1 -- 172.21.15.178:0/3643388220 shutdown_connections 2023-11-15T04:58:28.900 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.898+0000 7f430015b700 1 --2- 172.21.15.178:0/3643388220 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f42e402ff60 0x7f42e4032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:28.901 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.898+0000 7f430015b700 1 --2- 172.21.15.178:0/3643388220 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42f80f5410 0x7f42f8109430 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:28.901 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.898+0000 7f430015b700 1 -- 172.21.15.178:0/3643388220 >> 172.21.15.178:0/3643388220 conn(0x7f42f80f0b70 msgr2=0x7f42f80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:28.901 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.898+0000 7f430015b700 1 -- 172.21.15.178:0/3643388220 shutdown_connections 2023-11-15T04:58:28.901 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:28.898+0000 7f430015b700 1 -- 172.21.15.178:0/3643388220 wait complete. 2023-11-15T04:58:28.902 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:58:29.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:28 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:28.897734+0000 mon.smithi163 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.178:0/3643388220' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:58:30.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:29 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:28.336675+0000 mgr.smithi163.qesyrx (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:30.311 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:58:30.312 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:31.460 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.458+0000 7ff2b77fb700 1 -- 172.21.15.178:0/2173078768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2b00601e0 msgr2=0x7ff2b00f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:31.460 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.458+0000 7ff2b77fb700 1 --2- 172.21.15.178:0/2173078768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2b00601e0 0x7ff2b00f9190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7ff2a0009fb0 tx=0x7ff2a0009b30).stop 2023-11-15T04:58:31.461 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.458+0000 7ff2b77fb700 1 -- 172.21.15.178:0/2173078768 shutdown_connections 2023-11-15T04:58:31.461 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.458+0000 7ff2b77fb700 1 --2- 172.21.15.178:0/2173078768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2b00601e0 0x7ff2b00f9190 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:31.461 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.458+0000 7ff2b77fb700 1 -- 172.21.15.178:0/2173078768 >> 172.21.15.178:0/2173078768 conn(0x7ff2b00f0b70 msgr2=0x7ff2b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:31.461 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.458+0000 7ff2b77fb700 1 -- 172.21.15.178:0/2173078768 shutdown_connections 2023-11-15T04:58:31.461 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.458+0000 7ff2b77fb700 1 -- 172.21.15.178:0/2173078768 wait complete. 2023-11-15T04:58:31.462 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.459+0000 7ff2b77fb700 1 Processor -- start 2023-11-15T04:58:31.462 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.459+0000 7ff2b77fb700 1 -- start start 2023-11-15T04:58:31.462 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.459+0000 7ff2b77fb700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2b00601e0 0x7ff2b01066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:31.462 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.459+0000 7ff2b77fb700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b0106ba0 con 0x7ff2b00601e0 2023-11-15T04:58:31.463 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.459+0000 7ff2b5597700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2b00601e0 0x7ff2b01066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:31.463 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.459+0000 7ff2b5597700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2b00601e0 0x7ff2b01066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:39832/0 (socket says 172.21.15.178:39832) 2023-11-15T04:58:31.463 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.459+0000 7ff2b5597700 1 -- 172.21.15.178:0/2397008219 learned_addr learned my addr 172.21.15.178:0/2397008219 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:31.464 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.459+0000 7ff2b5597700 1 -- 172.21.15.178:0/2397008219 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a0005040 con 0x7ff2b00601e0 2023-11-15T04:58:31.465 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.459+0000 7ff2b5597700 1 --2- 172.21.15.178:0/2397008219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2b00601e0 0x7ff2b01066b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7ff2a0004dd0 tx=0x7ff2a0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:31.465 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.460+0000 7ff2a5ffb700 1 -- 172.21.15.178:0/2397008219 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2a0018070 con 0x7ff2b00601e0 2023-11-15T04:58:31.465 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.460+0000 7ff2b77fb700 1 -- 172.21.15.178:0/2397008219 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b0106da0 con 0x7ff2b00601e0 2023-11-15T04:58:31.465 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.460+0000 7ff2a5ffb700 1 -- 172.21.15.178:0/2397008219 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2a0013410 con 0x7ff2b00601e0 2023-11-15T04:58:31.466 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.460+0000 7ff2a5ffb700 1 -- 172.21.15.178:0/2397008219 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2a001c3c0 con 0x7ff2b00601e0 2023-11-15T04:58:31.466 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.460+0000 7ff2b77fb700 1 -- 172.21.15.178:0/2397008219 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b0107230 con 0x7ff2b00601e0 2023-11-15T04:58:31.466 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.461+0000 7ff2a5ffb700 1 -- 172.21.15.178:0/2397008219 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2a001d420 con 0x7ff2b00601e0 2023-11-15T04:58:31.466 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.461+0000 7ff2b77fb700 1 -- 172.21.15.178:0/2397008219 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2b0045ad0 con 0x7ff2b00601e0 2023-11-15T04:58:31.466 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.461+0000 7ff2a5ffb700 1 --2- 172.21.15.178:0/2397008219 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff29c02ff60 0x7ff29c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:31.467 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.461+0000 7ff2a5ffb700 1 -- 172.21.15.178:0/2397008219 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff2a0041450 con 0x7ff2b00601e0 2023-11-15T04:58:31.467 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.461+0000 7ff2b4d96700 1 --2- 172.21.15.178:0/2397008219 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff29c02ff60 0x7ff29c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:31.467 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.461+0000 7ff2b4d96700 1 --2- 172.21.15.178:0/2397008219 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff29c02ff60 0x7ff29c032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7ff2ac0096a0 tx=0x7ff2ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:31.467 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.466+0000 7ff2a5ffb700 1 -- 172.21.15.178:0/2397008219 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2a001d820 con 0x7ff2b00601e0 2023-11-15T04:58:31.673 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.671+0000 7ff2b77fb700 1 -- 172.21.15.178:0/2397008219 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2b01080e0 con 0x7ff2b00601e0 2023-11-15T04:58:31.673 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.672+0000 7ff2a5ffb700 1 -- 172.21.15.178:0/2397008219 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff2a001b370 con 0x7ff2b00601e0 2023-11-15T04:58:31.674 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:31.674 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:31.675 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.674+0000 7ff2b77fb700 1 -- 172.21.15.178:0/2397008219 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff29c02ff60 msgr2=0x7ff29c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:31.676 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.674+0000 7ff2b77fb700 1 --2- 172.21.15.178:0/2397008219 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff29c02ff60 0x7ff29c032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7ff2ac0096a0 tx=0x7ff2ac006b40).stop 2023-11-15T04:58:31.676 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.674+0000 7ff2b77fb700 1 -- 172.21.15.178:0/2397008219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2b00601e0 msgr2=0x7ff2b01066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:31.676 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.674+0000 7ff2b77fb700 1 --2- 172.21.15.178:0/2397008219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2b00601e0 0x7ff2b01066b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7ff2a0004dd0 tx=0x7ff2a0009b30).stop 2023-11-15T04:58:31.676 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.674+0000 7ff2b77fb700 1 -- 172.21.15.178:0/2397008219 shutdown_connections 2023-11-15T04:58:31.676 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.674+0000 7ff2b77fb700 1 --2- 172.21.15.178:0/2397008219 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff29c02ff60 0x7ff29c032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:31.677 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.674+0000 7ff2b77fb700 1 --2- 172.21.15.178:0/2397008219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2b00601e0 0x7ff2b01066b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:31.677 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.674+0000 7ff2b77fb700 1 -- 172.21.15.178:0/2397008219 >> 172.21.15.178:0/2397008219 conn(0x7ff2b00f0b70 msgr2=0x7ff2b00f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:31.677 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.674+0000 7ff2b77fb700 1 -- 172.21.15.178:0/2397008219 shutdown_connections 2023-11-15T04:58:31.677 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:31.674+0000 7ff2b77fb700 1 -- 172.21.15.178:0/2397008219 wait complete. 2023-11-15T04:58:31.677 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:58:32.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:31 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:30.337007+0000 mgr.smithi163.qesyrx (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:32.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:31 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:31.673858+0000 mon.smithi163 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.178:0/2397008219' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:58:33.049 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:58:33.050 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:34.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:33 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:32.337377+0000 mgr.smithi163.qesyrx (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:34.228 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.225+0000 7f00d47b8700 1 -- 172.21.15.178:0/1378232115 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00cc060280 msgr2=0x7f00cc0fb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:34.229 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.225+0000 7f00d47b8700 1 --2- 172.21.15.178:0/1378232115 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00cc060280 0x7f00cc0fb4f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f00bc009fb0 tx=0x7f00bc009b30).stop 2023-11-15T04:58:34.229 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.225+0000 7f00d47b8700 1 -- 172.21.15.178:0/1378232115 shutdown_connections 2023-11-15T04:58:34.229 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.225+0000 7f00d47b8700 1 --2- 172.21.15.178:0/1378232115 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00cc060280 0x7f00cc0fb4f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:34.229 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.225+0000 7f00d47b8700 1 -- 172.21.15.178:0/1378232115 >> 172.21.15.178:0/1378232115 conn(0x7f00cc0f0b70 msgr2=0x7f00cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:34.230 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.225+0000 7f00d47b8700 1 -- 172.21.15.178:0/1378232115 shutdown_connections 2023-11-15T04:58:34.230 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.225+0000 7f00d47b8700 1 -- 172.21.15.178:0/1378232115 wait complete. 2023-11-15T04:58:34.230 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.226+0000 7f00d47b8700 1 Processor -- start 2023-11-15T04:58:34.231 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.226+0000 7f00d47b8700 1 -- start start 2023-11-15T04:58:34.231 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.226+0000 7f00d47b8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00cc060280 0x7f00cc1088e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:34.231 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.226+0000 7f00d47b8700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00cc108dd0 con 0x7f00cc060280 2023-11-15T04:58:34.231 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.226+0000 7f00d2554700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00cc060280 0x7f00cc1088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:34.232 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.226+0000 7f00d2554700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00cc060280 0x7f00cc1088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:39846/0 (socket says 172.21.15.178:39846) 2023-11-15T04:58:34.232 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.226+0000 7f00d2554700 1 -- 172.21.15.178:0/3727264060 learned_addr learned my addr 172.21.15.178:0/3727264060 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:34.232 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.227+0000 7f00d2554700 1 -- 172.21.15.178:0/3727264060 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00bc005040 con 0x7f00cc060280 2023-11-15T04:58:34.232 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.227+0000 7f00d2554700 1 --2- 172.21.15.178:0/3727264060 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00cc060280 0x7f00cc1088e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f00bc013510 tx=0x7f00bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:34.233 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.227+0000 7f00c2ffd700 1 -- 172.21.15.178:0/3727264060 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00bc009320 con 0x7f00cc060280 2023-11-15T04:58:34.233 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.227+0000 7f00d47b8700 1 -- 172.21.15.178:0/3727264060 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00cc108fd0 con 0x7f00cc060280 2023-11-15T04:58:34.233 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.227+0000 7f00c2ffd700 1 -- 172.21.15.178:0/3727264060 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f00bc013f00 con 0x7f00cc060280 2023-11-15T04:58:34.233 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.227+0000 7f00c2ffd700 1 -- 172.21.15.178:0/3727264060 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00bc014dc0 con 0x7f00cc060280 2023-11-15T04:58:34.233 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.227+0000 7f00d47b8700 1 -- 172.21.15.178:0/3727264060 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00cc109460 con 0x7f00cc060280 2023-11-15T04:58:34.235 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.228+0000 7f00c2ffd700 1 -- 172.21.15.178:0/3727264060 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f00bc01d3d0 con 0x7f00cc060280 2023-11-15T04:58:34.235 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.228+0000 7f00c2ffd700 1 --2- 172.21.15.178:0/3727264060 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f00b802ff60 0x7f00b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:34.235 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.228+0000 7f00c2ffd700 1 -- 172.21.15.178:0/3727264060 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f00bc041030 con 0x7f00cc060280 2023-11-15T04:58:34.236 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.228+0000 7f00d47b8700 1 -- 172.21.15.178:0/3727264060 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00cc045ad0 con 0x7f00cc060280 2023-11-15T04:58:34.236 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.228+0000 7f00d1d53700 1 --2- 172.21.15.178:0/3727264060 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f00b802ff60 0x7f00b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:34.236 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.228+0000 7f00d1d53700 1 --2- 172.21.15.178:0/3727264060 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f00b802ff60 0x7f00b8032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f00c80096a0 tx=0x7f00c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:34.237 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.233+0000 7f00c2ffd700 1 -- 172.21.15.178:0/3727264060 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f00bc012920 con 0x7f00cc060280 2023-11-15T04:58:34.439 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.437+0000 7f00d47b8700 1 -- 172.21.15.178:0/3727264060 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f00cc10a180 con 0x7f00cc060280 2023-11-15T04:58:34.439 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.438+0000 7f00c2ffd700 1 -- 172.21.15.178:0/3727264060 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f00bc012330 con 0x7f00cc060280 2023-11-15T04:58:34.440 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:34.440 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:34.442 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.440+0000 7f00d47b8700 1 -- 172.21.15.178:0/3727264060 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f00b802ff60 msgr2=0x7f00b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:34.442 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.440+0000 7f00d47b8700 1 --2- 172.21.15.178:0/3727264060 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f00b802ff60 0x7f00b8032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f00c80096a0 tx=0x7f00c8006b40).stop 2023-11-15T04:58:34.442 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.440+0000 7f00d47b8700 1 -- 172.21.15.178:0/3727264060 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00cc060280 msgr2=0x7f00cc1088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:34.442 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.440+0000 7f00d47b8700 1 --2- 172.21.15.178:0/3727264060 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00cc060280 0x7f00cc1088e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f00bc013510 tx=0x7f00bc00b040).stop 2023-11-15T04:58:34.442 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.440+0000 7f00d47b8700 1 -- 172.21.15.178:0/3727264060 shutdown_connections 2023-11-15T04:58:34.443 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.440+0000 7f00d47b8700 1 --2- 172.21.15.178:0/3727264060 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f00b802ff60 0x7f00b8032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:34.443 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.440+0000 7f00d47b8700 1 --2- 172.21.15.178:0/3727264060 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00cc060280 0x7f00cc1088e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:34.443 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.440+0000 7f00d47b8700 1 -- 172.21.15.178:0/3727264060 >> 172.21.15.178:0/3727264060 conn(0x7f00cc0f0b70 msgr2=0x7f00cc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:34.443 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.440+0000 7f00d47b8700 1 -- 172.21.15.178:0/3727264060 shutdown_connections 2023-11-15T04:58:34.443 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:34.440+0000 7f00d47b8700 1 -- 172.21.15.178:0/3727264060 wait complete. 2023-11-15T04:58:34.444 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:58:35.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:34 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:34.439858+0000 mon.smithi163 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.178:0/3727264060' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:58:35.829 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:58:35.829 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:36.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:34.337705+0000 mgr.smithi163.qesyrx (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:36.976 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.973+0000 7f6c07e1c700 1 -- 172.21.15.178:0/2419146334 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c000f5410 msgr2=0x7f6c000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:36.976 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.973+0000 7f6c07e1c700 1 --2- 172.21.15.178:0/2419146334 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c000f5410 0x7f6c000f5830 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f6bf0009fb0 tx=0x7f6bf0009b30).stop 2023-11-15T04:58:36.976 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.973+0000 7f6c07e1c700 1 -- 172.21.15.178:0/2419146334 shutdown_connections 2023-11-15T04:58:36.977 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.973+0000 7f6c07e1c700 1 --2- 172.21.15.178:0/2419146334 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c000f5410 0x7f6c000f5830 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:36.977 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.973+0000 7f6c07e1c700 1 -- 172.21.15.178:0/2419146334 >> 172.21.15.178:0/2419146334 conn(0x7f6c000f0b70 msgr2=0x7f6c000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:36.977 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.973+0000 7f6c07e1c700 1 -- 172.21.15.178:0/2419146334 shutdown_connections 2023-11-15T04:58:36.977 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.973+0000 7f6c07e1c700 1 -- 172.21.15.178:0/2419146334 wait complete. 2023-11-15T04:58:36.977 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.974+0000 7f6c07e1c700 1 Processor -- start 2023-11-15T04:58:36.978 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.974+0000 7f6c07e1c700 1 -- start start 2023-11-15T04:58:36.978 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.974+0000 7f6c07e1c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c000f5410 0x7f6c00109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:36.978 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.974+0000 7f6c07e1c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c00060220 con 0x7f6c000f5410 2023-11-15T04:58:36.978 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.974+0000 7f6c05bb8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c000f5410 0x7f6c00109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:36.978 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.974+0000 7f6c05bb8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c000f5410 0x7f6c00109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:51378/0 (socket says 172.21.15.178:51378) 2023-11-15T04:58:36.979 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.974+0000 7f6c05bb8700 1 -- 172.21.15.178:0/2397841969 learned_addr learned my addr 172.21.15.178:0/2397841969 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:36.979 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.974+0000 7f6c05bb8700 1 -- 172.21.15.178:0/2397841969 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf0005040 con 0x7f6c000f5410 2023-11-15T04:58:36.979 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.975+0000 7f6c05bb8700 1 --2- 172.21.15.178:0/2397841969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c000f5410 0x7f6c00109450 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f6bf0014000 tx=0x7f6bf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:36.979 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.975+0000 7f6bf67fc700 1 -- 172.21.15.178:0/2397841969 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bf00095f0 con 0x7f6c000f5410 2023-11-15T04:58:36.979 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.975+0000 7f6bf67fc700 1 -- 172.21.15.178:0/2397841969 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6bf0007e70 con 0x7f6c000f5410 2023-11-15T04:58:36.980 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.975+0000 7f6bf67fc700 1 -- 172.21.15.178:0/2397841969 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bf0014ea0 con 0x7f6c000f5410 2023-11-15T04:58:36.980 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.975+0000 7f6c07e1c700 1 -- 172.21.15.178:0/2397841969 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c00060420 con 0x7f6c000f5410 2023-11-15T04:58:36.980 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.975+0000 7f6c07e1c700 1 -- 172.21.15.178:0/2397841969 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c00109b00 con 0x7f6c000f5410 2023-11-15T04:58:36.981 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.975+0000 7f6bf67fc700 1 -- 172.21.15.178:0/2397841969 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6bf0025070 con 0x7f6c000f5410 2023-11-15T04:58:36.982 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.976+0000 7f6c07e1c700 1 -- 172.21.15.178:0/2397841969 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c00059100 con 0x7f6c000f5410 2023-11-15T04:58:36.982 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.976+0000 7f6bf67fc700 1 --2- 172.21.15.178:0/2397841969 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6bec02ffb0 0x7f6bec032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:36.982 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.976+0000 7f6bf67fc700 1 -- 172.21.15.178:0/2397841969 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6bf0040900 con 0x7f6c000f5410 2023-11-15T04:58:36.982 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.976+0000 7f6c053b7700 1 --2- 172.21.15.178:0/2397841969 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6bec02ffb0 0x7f6bec032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:36.982 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.976+0000 7f6c053b7700 1 --2- 172.21.15.178:0/2397841969 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6bec02ffb0 0x7f6bec032470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f6bfc0096a0 tx=0x7f6bfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:36.985 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:36.981+0000 7f6bf67fc700 1 -- 172.21.15.178:0/2397841969 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6bf00079c0 con 0x7f6c000f5410 2023-11-15T04:58:37.189 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:37.187+0000 7f6c07e1c700 1 -- 172.21.15.178:0/2397841969 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c00045ad0 con 0x7f6c000f5410 2023-11-15T04:58:37.190 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:37.188+0000 7f6bf67fc700 1 -- 172.21.15.178:0/2397841969 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6bf000f300 con 0x7f6c000f5410 2023-11-15T04:58:37.190 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:37.190 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:37.191 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:37.190+0000 7f6c07e1c700 1 -- 172.21.15.178:0/2397841969 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6bec02ffb0 msgr2=0x7f6bec032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:37.191 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:37.190+0000 7f6c07e1c700 1 --2- 172.21.15.178:0/2397841969 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6bec02ffb0 0x7f6bec032470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f6bfc0096a0 tx=0x7f6bfc006b40).stop 2023-11-15T04:58:37.192 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:37.190+0000 7f6c07e1c700 1 -- 172.21.15.178:0/2397841969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c000f5410 msgr2=0x7f6c00109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:37.192 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:37.190+0000 7f6c07e1c700 1 --2- 172.21.15.178:0/2397841969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c000f5410 0x7f6c00109450 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f6bf0014000 tx=0x7f6bf000b040).stop 2023-11-15T04:58:37.192 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:37.190+0000 7f6c07e1c700 1 -- 172.21.15.178:0/2397841969 shutdown_connections 2023-11-15T04:58:37.192 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:37.190+0000 7f6c07e1c700 1 --2- 172.21.15.178:0/2397841969 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6bec02ffb0 0x7f6bec032470 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:37.192 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:37.190+0000 7f6c07e1c700 1 --2- 172.21.15.178:0/2397841969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c000f5410 0x7f6c00109450 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:37.193 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:37.190+0000 7f6c07e1c700 1 -- 172.21.15.178:0/2397841969 >> 172.21.15.178:0/2397841969 conn(0x7f6c000f0b70 msgr2=0x7f6c000f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:37.193 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:37.190+0000 7f6c07e1c700 1 -- 172.21.15.178:0/2397841969 shutdown_connections 2023-11-15T04:58:37.193 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:37.190+0000 7f6c07e1c700 1 -- 172.21.15.178:0/2397841969 wait complete. 2023-11-15T04:58:37.193 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:58:38.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:37 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:36.338044+0000 mgr.smithi163.qesyrx (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:38.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:37 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:37.189456+0000 mon.smithi163 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.178:0/2397841969' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:58:38.603 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:58:38.604 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:39.768 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.764+0000 7f7e73fff700 1 -- 172.21.15.178:0/3439481367 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e740f62a0 msgr2=0x7f7e740f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:39.768 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.764+0000 7f7e73fff700 1 --2- 172.21.15.178:0/3439481367 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e740f62a0 0x7f7e740f66c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f7e64009fb0 tx=0x7f7e64009b30).stop 2023-11-15T04:58:39.769 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.765+0000 7f7e73fff700 1 -- 172.21.15.178:0/3439481367 shutdown_connections 2023-11-15T04:58:39.769 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.765+0000 7f7e73fff700 1 --2- 172.21.15.178:0/3439481367 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e740f62a0 0x7f7e740f66c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:39.769 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.765+0000 7f7e73fff700 1 -- 172.21.15.178:0/3439481367 >> 172.21.15.178:0/3439481367 conn(0x7f7e740f1aa0 msgr2=0x7f7e740f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:39.769 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.765+0000 7f7e73fff700 1 -- 172.21.15.178:0/3439481367 shutdown_connections 2023-11-15T04:58:39.770 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.765+0000 7f7e73fff700 1 -- 172.21.15.178:0/3439481367 wait complete. 2023-11-15T04:58:39.770 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.765+0000 7f7e73fff700 1 Processor -- start 2023-11-15T04:58:39.770 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.765+0000 7f7e73fff700 1 -- start start 2023-11-15T04:58:39.771 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.765+0000 7f7e73fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e740f62a0 0x7f7e741086b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:39.771 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.765+0000 7f7e73fff700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e74108ba0 con 0x7f7e740f62a0 2023-11-15T04:58:39.771 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.766+0000 7f7e72ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e740f62a0 0x7f7e741086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:39.771 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.766+0000 7f7e72ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e740f62a0 0x7f7e741086b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:51386/0 (socket says 172.21.15.178:51386) 2023-11-15T04:58:39.772 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.766+0000 7f7e72ffd700 1 -- 172.21.15.178:0/920378130 learned_addr learned my addr 172.21.15.178:0/920378130 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:39.772 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.766+0000 7f7e72ffd700 1 -- 172.21.15.178:0/920378130 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e64005040 con 0x7f7e740f62a0 2023-11-15T04:58:39.772 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.766+0000 7f7e72ffd700 1 --2- 172.21.15.178:0/920378130 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e740f62a0 0x7f7e741086b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f7e64013510 tx=0x7f7e6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:39.772 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.766+0000 7f7e5b7fe700 1 -- 172.21.15.178:0/920378130 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e64009320 con 0x7f7e740f62a0 2023-11-15T04:58:39.773 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.766+0000 7f7e5b7fe700 1 -- 172.21.15.178:0/920378130 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e64013f00 con 0x7f7e740f62a0 2023-11-15T04:58:39.773 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.766+0000 7f7e73fff700 1 -- 172.21.15.178:0/920378130 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e74108da0 con 0x7f7e740f62a0 2023-11-15T04:58:39.773 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.766+0000 7f7e5b7fe700 1 -- 172.21.15.178:0/920378130 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e64014dc0 con 0x7f7e740f62a0 2023-11-15T04:58:39.773 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.766+0000 7f7e73fff700 1 -- 172.21.15.178:0/920378130 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e74109230 con 0x7f7e740f62a0 2023-11-15T04:58:39.773 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.767+0000 7f7e5b7fe700 1 -- 172.21.15.178:0/920378130 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7e640095f0 con 0x7f7e740f62a0 2023-11-15T04:58:39.773 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.767+0000 7f7e73fff700 1 -- 172.21.15.178:0/920378130 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e74045ad0 con 0x7f7e740f62a0 2023-11-15T04:58:39.774 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.767+0000 7f7e5b7fe700 1 --2- 172.21.15.178:0/920378130 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7e5c02ff60 0x7f7e5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:39.775 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.767+0000 7f7e5b7fe700 1 -- 172.21.15.178:0/920378130 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7e640410c0 con 0x7f7e740f62a0 2023-11-15T04:58:39.775 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.767+0000 7f7e727fc700 1 --2- 172.21.15.178:0/920378130 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7e5c02ff60 0x7f7e5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:39.775 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.768+0000 7f7e727fc700 1 --2- 172.21.15.178:0/920378130 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7e5c02ff60 0x7f7e5c032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f7e680096a0 tx=0x7f7e68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:39.775 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.773+0000 7f7e5b7fe700 1 -- 172.21.15.178:0/920378130 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e6401c790 con 0x7f7e740f62a0 2023-11-15T04:58:39.980 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.979+0000 7f7e73fff700 1 -- 172.21.15.178:0/920378130 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e7410a0e0 con 0x7f7e740f62a0 2023-11-15T04:58:39.981 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.981+0000 7f7e5b7fe700 1 -- 172.21.15.178:0/920378130 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7e64010230 con 0x7f7e740f62a0 2023-11-15T04:58:39.981 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:39.982 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:39.983 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.981+0000 7f7e73fff700 1 -- 172.21.15.178:0/920378130 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7e5c02ff60 msgr2=0x7f7e5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:39.983 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.981+0000 7f7e73fff700 1 --2- 172.21.15.178:0/920378130 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7e5c02ff60 0x7f7e5c032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f7e680096a0 tx=0x7f7e68006b40).stop 2023-11-15T04:58:39.983 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.981+0000 7f7e73fff700 1 -- 172.21.15.178:0/920378130 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e740f62a0 msgr2=0x7f7e741086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:39.983 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.981+0000 7f7e73fff700 1 --2- 172.21.15.178:0/920378130 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e740f62a0 0x7f7e741086b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f7e64013510 tx=0x7f7e6400b040).stop 2023-11-15T04:58:39.984 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.981+0000 7f7e73fff700 1 -- 172.21.15.178:0/920378130 shutdown_connections 2023-11-15T04:58:39.984 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.981+0000 7f7e73fff700 1 --2- 172.21.15.178:0/920378130 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7e5c02ff60 0x7f7e5c032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:39.984 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.981+0000 7f7e73fff700 1 --2- 172.21.15.178:0/920378130 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e740f62a0 0x7f7e741086b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:39.984 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.981+0000 7f7e73fff700 1 -- 172.21.15.178:0/920378130 >> 172.21.15.178:0/920378130 conn(0x7f7e740f1aa0 msgr2=0x7f7e740f24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:39.984 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.981+0000 7f7e73fff700 1 -- 172.21.15.178:0/920378130 shutdown_connections 2023-11-15T04:58:39.984 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:39.981+0000 7f7e73fff700 1 -- 172.21.15.178:0/920378130 wait complete. 2023-11-15T04:58:39.985 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:58:40.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:39 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:38.338327+0000 mgr.smithi163.qesyrx (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:41.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:40 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:39.981455+0000 mon.smithi163 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.178:0/920378130' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:58:41.365 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:58:41.366 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:42.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:41 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:40.338563+0000 mgr.smithi163.qesyrx (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:42.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.527+0000 7f5ab2885700 1 -- 172.21.15.178:0/1146470575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5aac0f6510 msgr2=0x7f5aac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:42.528 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.527+0000 7f5ab2885700 1 --2- 172.21.15.178:0/1146470575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5aac0f6510 0x7f5aac0f6930 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f5aa0009fb0 tx=0x7f5aa0009b30).stop 2023-11-15T04:58:42.528 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.527+0000 7f5ab2885700 1 -- 172.21.15.178:0/1146470575 shutdown_connections 2023-11-15T04:58:42.528 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.527+0000 7f5ab2885700 1 --2- 172.21.15.178:0/1146470575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5aac0f6510 0x7f5aac0f6930 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:42.528 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.527+0000 7f5ab2885700 1 -- 172.21.15.178:0/1146470575 >> 172.21.15.178:0/1146470575 conn(0x7f5aac0f1c70 msgr2=0x7f5aac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:42.529 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.527+0000 7f5ab2885700 1 -- 172.21.15.178:0/1146470575 shutdown_connections 2023-11-15T04:58:42.529 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.527+0000 7f5ab2885700 1 -- 172.21.15.178:0/1146470575 wait complete. 2023-11-15T04:58:42.531 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.527+0000 7f5ab2885700 1 Processor -- start 2023-11-15T04:58:42.531 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.528+0000 7f5ab2885700 1 -- start start 2023-11-15T04:58:42.531 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.528+0000 7f5ab2885700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5aac0f6510 0x7f5aac106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:42.532 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.528+0000 7f5ab2885700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aac106c20 con 0x7f5aac0f6510 2023-11-15T04:58:42.532 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.528+0000 7f5aabfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5aac0f6510 0x7f5aac106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:42.532 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.528+0000 7f5aabfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5aac0f6510 0x7f5aac106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:51396/0 (socket says 172.21.15.178:51396) 2023-11-15T04:58:42.532 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.528+0000 7f5aabfff700 1 -- 172.21.15.178:0/3503674332 learned_addr learned my addr 172.21.15.178:0/3503674332 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:42.532 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.528+0000 7f5aabfff700 1 -- 172.21.15.178:0/3503674332 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa0005040 con 0x7f5aac0f6510 2023-11-15T04:58:42.532 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.528+0000 7f5aabfff700 1 --2- 172.21.15.178:0/3503674332 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5aac0f6510 0x7f5aac106730 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f5aa0014000 tx=0x7f5aa000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:42.533 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.529+0000 7f5aa8ff9700 1 -- 172.21.15.178:0/3503674332 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aa00095f0 con 0x7f5aac0f6510 2023-11-15T04:58:42.533 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.529+0000 7f5ab2885700 1 -- 172.21.15.178:0/3503674332 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aac106e20 con 0x7f5aac0f6510 2023-11-15T04:58:42.533 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.529+0000 7f5aa8ff9700 1 -- 172.21.15.178:0/3503674332 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5aa0007e70 con 0x7f5aac0f6510 2023-11-15T04:58:42.533 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.529+0000 7f5aa8ff9700 1 -- 172.21.15.178:0/3503674332 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aa0014ea0 con 0x7f5aac0f6510 2023-11-15T04:58:42.533 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.529+0000 7f5ab2885700 1 -- 172.21.15.178:0/3503674332 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aac1072b0 con 0x7f5aac0f6510 2023-11-15T04:58:42.534 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.529+0000 7f5aa8ff9700 1 -- 172.21.15.178:0/3503674332 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5aa0025070 con 0x7f5aac0f6510 2023-11-15T04:58:42.534 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.530+0000 7f5ab2885700 1 -- 172.21.15.178:0/3503674332 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aac045ad0 con 0x7f5aac0f6510 2023-11-15T04:58:42.534 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.530+0000 7f5aa8ff9700 1 --2- 172.21.15.178:0/3503674332 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5a9402ff60 0x7f5a94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:42.534 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.530+0000 7f5aa8ff9700 1 -- 172.21.15.178:0/3503674332 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5aa00409f0 con 0x7f5aac0f6510 2023-11-15T04:58:42.534 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.530+0000 7f5aab7fe700 1 --2- 172.21.15.178:0/3503674332 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5a9402ff60 0x7f5a94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:42.535 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.530+0000 7f5aab7fe700 1 --2- 172.21.15.178:0/3503674332 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5a9402ff60 0x7f5a94032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f5a9c0096a0 tx=0x7f5a9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:42.535 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.535+0000 7f5aa8ff9700 1 -- 172.21.15.178:0/3503674332 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5aa00079c0 con 0x7f5aac0f6510 2023-11-15T04:58:42.740 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.740+0000 7f5ab2885700 1 -- 172.21.15.178:0/3503674332 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5aac0fbf60 con 0x7f5aac0f6510 2023-11-15T04:58:42.754 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.740+0000 7f5aa8ff9700 1 -- 172.21.15.178:0/3503674332 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5aa000f300 con 0x7f5aac0f6510 2023-11-15T04:58:42.754 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:42.755 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:42.756 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.742+0000 7f5ab2885700 1 -- 172.21.15.178:0/3503674332 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5a9402ff60 msgr2=0x7f5a94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:42.756 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.742+0000 7f5ab2885700 1 --2- 172.21.15.178:0/3503674332 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5a9402ff60 0x7f5a94032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f5a9c0096a0 tx=0x7f5a9c006b40).stop 2023-11-15T04:58:42.756 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.742+0000 7f5ab2885700 1 -- 172.21.15.178:0/3503674332 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5aac0f6510 msgr2=0x7f5aac106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:42.756 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.742+0000 7f5ab2885700 1 --2- 172.21.15.178:0/3503674332 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5aac0f6510 0x7f5aac106730 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f5aa0014000 tx=0x7f5aa000b040).stop 2023-11-15T04:58:42.756 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.742+0000 7f5ab2885700 1 -- 172.21.15.178:0/3503674332 shutdown_connections 2023-11-15T04:58:42.757 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.742+0000 7f5ab2885700 1 --2- 172.21.15.178:0/3503674332 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5a9402ff60 0x7f5a94032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:42.757 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.742+0000 7f5ab2885700 1 --2- 172.21.15.178:0/3503674332 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5aac0f6510 0x7f5aac106730 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:42.757 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.742+0000 7f5ab2885700 1 -- 172.21.15.178:0/3503674332 >> 172.21.15.178:0/3503674332 conn(0x7f5aac0f1c70 msgr2=0x7f5aac0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:42.757 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.742+0000 7f5ab2885700 1 -- 172.21.15.178:0/3503674332 shutdown_connections 2023-11-15T04:58:42.757 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:42.742+0000 7f5ab2885700 1 -- 172.21.15.178:0/3503674332 wait complete. 2023-11-15T04:58:42.758 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:58:43.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:42 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:42.741040+0000 mon.smithi163 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.178:0/3503674332' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:58:44.156 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:58:44.157 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:44.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:43 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:42.338852+0000 mgr.smithi163.qesyrx (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:45.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.339+0000 7ff4151f4700 1 -- 172.21.15.178:0/756208257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4100f5410 msgr2=0x7ff4100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:45.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.339+0000 7ff4151f4700 1 --2- 172.21.15.178:0/756208257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4100f5410 0x7ff4100f5830 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7ff404009fb0 tx=0x7ff404009b30).stop 2023-11-15T04:58:45.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.339+0000 7ff4151f4700 1 -- 172.21.15.178:0/756208257 shutdown_connections 2023-11-15T04:58:45.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.339+0000 7ff4151f4700 1 --2- 172.21.15.178:0/756208257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4100f5410 0x7ff4100f5830 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:45.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.339+0000 7ff4151f4700 1 -- 172.21.15.178:0/756208257 >> 172.21.15.178:0/756208257 conn(0x7ff4100f0b70 msgr2=0x7ff4100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:45.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.339+0000 7ff4151f4700 1 -- 172.21.15.178:0/756208257 shutdown_connections 2023-11-15T04:58:45.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.339+0000 7ff4151f4700 1 -- 172.21.15.178:0/756208257 wait complete. 2023-11-15T04:58:45.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.340+0000 7ff4151f4700 1 Processor -- start 2023-11-15T04:58:45.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.340+0000 7ff4151f4700 1 -- start start 2023-11-15T04:58:45.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.340+0000 7ff4151f4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4100f5410 0x7ff410105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:45.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.340+0000 7ff4151f4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff410105b20 con 0x7ff4100f5410 2023-11-15T04:58:45.344 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.340+0000 7ff40ed9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4100f5410 0x7ff410105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:45.344 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.340+0000 7ff40ed9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4100f5410 0x7ff410105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:49532/0 (socket says 172.21.15.178:49532) 2023-11-15T04:58:45.344 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.340+0000 7ff40ed9d700 1 -- 172.21.15.178:0/2980261142 learned_addr learned my addr 172.21.15.178:0/2980261142 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:45.344 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.341+0000 7ff40ed9d700 1 -- 172.21.15.178:0/2980261142 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff404005040 con 0x7ff4100f5410 2023-11-15T04:58:45.344 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.341+0000 7ff40ed9d700 1 --2- 172.21.15.178:0/2980261142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4100f5410 0x7ff410105630 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7ff404013510 tx=0x7ff40400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:45.345 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.341+0000 7ff3f77fe700 1 -- 172.21.15.178:0/2980261142 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff404009320 con 0x7ff4100f5410 2023-11-15T04:58:45.345 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.341+0000 7ff4151f4700 1 -- 172.21.15.178:0/2980261142 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff410105d20 con 0x7ff4100f5410 2023-11-15T04:58:45.345 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.341+0000 7ff3f77fe700 1 -- 172.21.15.178:0/2980261142 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff404013f00 con 0x7ff4100f5410 2023-11-15T04:58:45.345 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.341+0000 7ff3f77fe700 1 -- 172.21.15.178:0/2980261142 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff404014d90 con 0x7ff4100f5410 2023-11-15T04:58:45.345 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.341+0000 7ff4151f4700 1 -- 172.21.15.178:0/2980261142 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4101061b0 con 0x7ff4100f5410 2023-11-15T04:58:45.346 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.342+0000 7ff3f77fe700 1 -- 172.21.15.178:0/2980261142 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff4040095f0 con 0x7ff4100f5410 2023-11-15T04:58:45.346 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.342+0000 7ff4151f4700 1 -- 172.21.15.178:0/2980261142 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff410045ad0 con 0x7ff4100f5410 2023-11-15T04:58:45.346 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.342+0000 7ff3f77fe700 1 --2- 172.21.15.178:0/2980261142 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff3f802ff60 0x7ff3f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:45.346 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.342+0000 7ff3f77fe700 1 -- 172.21.15.178:0/2980261142 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff404040fe0 con 0x7ff4100f5410 2023-11-15T04:58:45.346 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.342+0000 7ff40e59c700 1 --2- 172.21.15.178:0/2980261142 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff3f802ff60 0x7ff3f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:45.347 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.342+0000 7ff40e59c700 1 --2- 172.21.15.178:0/2980261142 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff3f802ff60 0x7ff3f8032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7ff4000096a0 tx=0x7ff400006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:45.347 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.346+0000 7ff3f77fe700 1 -- 172.21.15.178:0/2980261142 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff404014420 con 0x7ff4100f5410 2023-11-15T04:58:45.552 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.552+0000 7ff4151f4700 1 -- 172.21.15.178:0/2980261142 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff4100fae60 con 0x7ff4100f5410 2023-11-15T04:58:45.553 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.553+0000 7ff3f77fe700 1 -- 172.21.15.178:0/2980261142 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff404010230 con 0x7ff4100f5410 2023-11-15T04:58:45.553 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:45.554 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:45.555 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.554+0000 7ff4151f4700 1 -- 172.21.15.178:0/2980261142 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff3f802ff60 msgr2=0x7ff3f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:45.555 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.554+0000 7ff4151f4700 1 --2- 172.21.15.178:0/2980261142 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff3f802ff60 0x7ff3f8032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7ff4000096a0 tx=0x7ff400006b40).stop 2023-11-15T04:58:45.555 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.554+0000 7ff4151f4700 1 -- 172.21.15.178:0/2980261142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4100f5410 msgr2=0x7ff410105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:45.555 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.554+0000 7ff4151f4700 1 --2- 172.21.15.178:0/2980261142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4100f5410 0x7ff410105630 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7ff404013510 tx=0x7ff40400b040).stop 2023-11-15T04:58:45.555 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.554+0000 7ff4151f4700 1 -- 172.21.15.178:0/2980261142 shutdown_connections 2023-11-15T04:58:45.556 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.554+0000 7ff4151f4700 1 --2- 172.21.15.178:0/2980261142 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff3f802ff60 0x7ff3f8032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:45.556 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.554+0000 7ff4151f4700 1 --2- 172.21.15.178:0/2980261142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4100f5410 0x7ff410105630 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:45.556 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.554+0000 7ff4151f4700 1 -- 172.21.15.178:0/2980261142 >> 172.21.15.178:0/2980261142 conn(0x7ff4100f0b70 msgr2=0x7ff4100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:45.556 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.554+0000 7ff4151f4700 1 -- 172.21.15.178:0/2980261142 shutdown_connections 2023-11-15T04:58:45.556 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:45.554+0000 7ff4151f4700 1 -- 172.21.15.178:0/2980261142 wait complete. 2023-11-15T04:58:45.557 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:58:46.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:45 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:44.339227+0000 mgr.smithi163.qesyrx (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:46.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:45 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:45.553571+0000 mon.smithi163 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.178:0/2980261142' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:58:46.934 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:58:46.935 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:48.168 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.168+0000 7fdb33fff700 1 -- 172.21.15.178:0/2466893556 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb340f62a0 msgr2=0x7fdb340f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:48.169 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.168+0000 7fdb33fff700 1 --2- 172.21.15.178:0/2466893556 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb340f62a0 0x7fdb340f66c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fdb28009fb0 tx=0x7fdb28009b30).stop 2023-11-15T04:58:48.169 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.168+0000 7fdb33fff700 1 -- 172.21.15.178:0/2466893556 shutdown_connections 2023-11-15T04:58:48.169 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.168+0000 7fdb33fff700 1 --2- 172.21.15.178:0/2466893556 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb340f62a0 0x7fdb340f66c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:48.169 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.168+0000 7fdb33fff700 1 -- 172.21.15.178:0/2466893556 >> 172.21.15.178:0/2466893556 conn(0x7fdb340f1aa0 msgr2=0x7fdb340f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:48.170 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.168+0000 7fdb33fff700 1 -- 172.21.15.178:0/2466893556 shutdown_connections 2023-11-15T04:58:48.170 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.168+0000 7fdb33fff700 1 -- 172.21.15.178:0/2466893556 wait complete. 2023-11-15T04:58:48.172 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.168+0000 7fdb33fff700 1 Processor -- start 2023-11-15T04:58:48.172 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.169+0000 7fdb33fff700 1 -- start start 2023-11-15T04:58:48.172 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.169+0000 7fdb33fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb340f62a0 0x7fdb341064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:48.172 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.169+0000 7fdb33fff700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb341069a0 con 0x7fdb340f62a0 2023-11-15T04:58:48.173 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.169+0000 7fdb32ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb340f62a0 0x7fdb341064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:48.173 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.169+0000 7fdb32ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb340f62a0 0x7fdb341064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:49554/0 (socket says 172.21.15.178:49554) 2023-11-15T04:58:48.173 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.169+0000 7fdb32ffd700 1 -- 172.21.15.178:0/3204754269 learned_addr learned my addr 172.21.15.178:0/3204754269 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:48.173 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.169+0000 7fdb32ffd700 1 -- 172.21.15.178:0/3204754269 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb28005040 con 0x7fdb340f62a0 2023-11-15T04:58:48.173 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.170+0000 7fdb32ffd700 1 --2- 172.21.15.178:0/3204754269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb340f62a0 0x7fdb341064b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fdb28013510 tx=0x7fdb2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:48.174 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.170+0000 7fdb1b7fe700 1 -- 172.21.15.178:0/3204754269 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb28009320 con 0x7fdb340f62a0 2023-11-15T04:58:48.174 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.170+0000 7fdb1b7fe700 1 -- 172.21.15.178:0/3204754269 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb28013f00 con 0x7fdb340f62a0 2023-11-15T04:58:48.174 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.170+0000 7fdb33fff700 1 -- 172.21.15.178:0/3204754269 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb34106ba0 con 0x7fdb340f62a0 2023-11-15T04:58:48.174 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.170+0000 7fdb1b7fe700 1 -- 172.21.15.178:0/3204754269 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb28014dc0 con 0x7fdb340f62a0 2023-11-15T04:58:48.174 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.170+0000 7fdb33fff700 1 -- 172.21.15.178:0/3204754269 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb34107030 con 0x7fdb340f62a0 2023-11-15T04:58:48.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.171+0000 7fdb1b7fe700 1 -- 172.21.15.178:0/3204754269 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdb280263a0 con 0x7fdb340f62a0 2023-11-15T04:58:48.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.171+0000 7fdb1b7fe700 1 --2- 172.21.15.178:0/3204754269 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdb1c02ffb0 0x7fdb1c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:48.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.171+0000 7fdb1b7fe700 1 -- 172.21.15.178:0/3204754269 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdb28040ff0 con 0x7fdb340f62a0 2023-11-15T04:58:48.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.171+0000 7fdb33fff700 1 -- 172.21.15.178:0/3204754269 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb34045ad0 con 0x7fdb340f62a0 2023-11-15T04:58:48.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.171+0000 7fdb327fc700 1 --2- 172.21.15.178:0/3204754269 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdb1c02ffb0 0x7fdb1c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:48.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.171+0000 7fdb327fc700 1 --2- 172.21.15.178:0/3204754269 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdb1c02ffb0 0x7fdb1c032470 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fdb240096a0 tx=0x7fdb24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:48.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.176+0000 7fdb1b7fe700 1 -- 172.21.15.178:0/3204754269 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdb28012920 con 0x7fdb340f62a0 2023-11-15T04:58:48.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:47 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:46.339581+0000 mgr.smithi163.qesyrx (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:48.380 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.380+0000 7fdb33fff700 1 -- 172.21.15.178:0/3204754269 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb34107d50 con 0x7fdb340f62a0 2023-11-15T04:58:48.381 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.381+0000 7fdb1b7fe700 1 -- 172.21.15.178:0/3204754269 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdb2802b020 con 0x7fdb340f62a0 2023-11-15T04:58:48.381 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:48.382 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:48.383 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.383+0000 7fdb33fff700 1 -- 172.21.15.178:0/3204754269 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdb1c02ffb0 msgr2=0x7fdb1c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:48.383 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.383+0000 7fdb33fff700 1 --2- 172.21.15.178:0/3204754269 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdb1c02ffb0 0x7fdb1c032470 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fdb240096a0 tx=0x7fdb24006b40).stop 2023-11-15T04:58:48.384 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.383+0000 7fdb33fff700 1 -- 172.21.15.178:0/3204754269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb340f62a0 msgr2=0x7fdb341064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:48.384 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.383+0000 7fdb33fff700 1 --2- 172.21.15.178:0/3204754269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb340f62a0 0x7fdb341064b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fdb28013510 tx=0x7fdb2800b040).stop 2023-11-15T04:58:48.384 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.383+0000 7fdb33fff700 1 -- 172.21.15.178:0/3204754269 shutdown_connections 2023-11-15T04:58:48.384 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.383+0000 7fdb33fff700 1 --2- 172.21.15.178:0/3204754269 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdb1c02ffb0 0x7fdb1c032470 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:48.384 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.383+0000 7fdb33fff700 1 --2- 172.21.15.178:0/3204754269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb340f62a0 0x7fdb341064b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:48.385 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.383+0000 7fdb33fff700 1 -- 172.21.15.178:0/3204754269 >> 172.21.15.178:0/3204754269 conn(0x7fdb340f1aa0 msgr2=0x7fdb340f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:48.385 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.383+0000 7fdb33fff700 1 -- 172.21.15.178:0/3204754269 shutdown_connections 2023-11-15T04:58:48.385 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:48.383+0000 7fdb33fff700 1 -- 172.21.15.178:0/3204754269 wait complete. 2023-11-15T04:58:48.386 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:58:49.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:48 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:48.381536+0000 mon.smithi163 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.178:0/3204754269' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:58:49.774 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:58:49.775 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:50.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:49 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:48.339896+0000 mgr.smithi163.qesyrx (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:50.956 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.954+0000 7f33a8d4f700 1 -- 172.21.15.178:0/3614988327 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33a40f5410 msgr2=0x7f33a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:50.956 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.954+0000 7f33a8d4f700 1 --2- 172.21.15.178:0/3614988327 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33a40f5410 0x7f33a40f5830 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f3398009fb0 tx=0x7f3398009b30).stop 2023-11-15T04:58:50.957 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.954+0000 7f33a8d4f700 1 -- 172.21.15.178:0/3614988327 shutdown_connections 2023-11-15T04:58:50.957 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.954+0000 7f33a8d4f700 1 --2- 172.21.15.178:0/3614988327 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33a40f5410 0x7f33a40f5830 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:50.957 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.954+0000 7f33a8d4f700 1 -- 172.21.15.178:0/3614988327 >> 172.21.15.178:0/3614988327 conn(0x7f33a40f0b70 msgr2=0x7f33a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:50.957 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.954+0000 7f33a8d4f700 1 -- 172.21.15.178:0/3614988327 shutdown_connections 2023-11-15T04:58:50.957 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.954+0000 7f33a8d4f700 1 -- 172.21.15.178:0/3614988327 wait complete. 2023-11-15T04:58:50.958 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.955+0000 7f33a8d4f700 1 Processor -- start 2023-11-15T04:58:50.958 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.955+0000 7f33a8d4f700 1 -- start start 2023-11-15T04:58:50.958 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.955+0000 7f33a8d4f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33a40f5410 0x7f33a4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:50.958 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.955+0000 7f33a8d4f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a4105b20 con 0x7f33a40f5410 2023-11-15T04:58:50.958 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.955+0000 7f33a259c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33a40f5410 0x7f33a4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:50.959 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.955+0000 7f33a259c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33a40f5410 0x7f33a4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:49576/0 (socket says 172.21.15.178:49576) 2023-11-15T04:58:50.959 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.955+0000 7f33a259c700 1 -- 172.21.15.178:0/1846454921 learned_addr learned my addr 172.21.15.178:0/1846454921 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:50.959 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.955+0000 7f33a259c700 1 -- 172.21.15.178:0/1846454921 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3398005040 con 0x7f33a40f5410 2023-11-15T04:58:50.959 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.955+0000 7f33a259c700 1 --2- 172.21.15.178:0/1846454921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33a40f5410 0x7f33a4105630 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f3398013510 tx=0x7f339800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:50.959 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.956+0000 7f3392ffd700 1 -- 172.21.15.178:0/1846454921 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3398009320 con 0x7f33a40f5410 2023-11-15T04:58:50.960 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.956+0000 7f3392ffd700 1 -- 172.21.15.178:0/1846454921 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3398013f00 con 0x7f33a40f5410 2023-11-15T04:58:50.960 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.956+0000 7f33a8d4f700 1 -- 172.21.15.178:0/1846454921 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a4105d20 con 0x7f33a40f5410 2023-11-15T04:58:50.960 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.956+0000 7f3392ffd700 1 -- 172.21.15.178:0/1846454921 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3398014d90 con 0x7f33a40f5410 2023-11-15T04:58:50.960 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.956+0000 7f33a8d4f700 1 -- 172.21.15.178:0/1846454921 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a41061b0 con 0x7f33a40f5410 2023-11-15T04:58:50.961 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.956+0000 7f3392ffd700 1 -- 172.21.15.178:0/1846454921 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f33980263a0 con 0x7f33a40f5410 2023-11-15T04:58:50.961 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.956+0000 7f33a8d4f700 1 -- 172.21.15.178:0/1846454921 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33a4045ad0 con 0x7f33a40f5410 2023-11-15T04:58:50.961 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.957+0000 7f3392ffd700 1 --2- 172.21.15.178:0/1846454921 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f338c02ff60 0x7f338c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:50.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.957+0000 7f3392ffd700 1 -- 172.21.15.178:0/1846454921 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3398040ff0 con 0x7f33a40f5410 2023-11-15T04:58:50.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.957+0000 7f33a1d9b700 1 --2- 172.21.15.178:0/1846454921 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f338c02ff60 0x7f338c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:50.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.957+0000 7f33a1d9b700 1 --2- 172.21.15.178:0/1846454921 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f338c02ff60 0x7f338c032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f33940096a0 tx=0x7f3394006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:50.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:50.961+0000 7f3392ffd700 1 -- 172.21.15.178:0/1846454921 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f339801db70 con 0x7f33a40f5410 2023-11-15T04:58:51.168 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:51.168+0000 7f33a8d4f700 1 -- 172.21.15.178:0/1846454921 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f33a40fae60 con 0x7f33a40f5410 2023-11-15T04:58:51.169 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:51.169+0000 7f3392ffd700 1 -- 172.21.15.178:0/1846454921 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f339801db70 con 0x7f33a40f5410 2023-11-15T04:58:51.169 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:51.169 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:51.171 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:51.170+0000 7f33a8d4f700 1 -- 172.21.15.178:0/1846454921 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f338c02ff60 msgr2=0x7f338c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:51.171 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:51.170+0000 7f33a8d4f700 1 --2- 172.21.15.178:0/1846454921 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f338c02ff60 0x7f338c032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f33940096a0 tx=0x7f3394006b40).stop 2023-11-15T04:58:51.171 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:51.170+0000 7f33a8d4f700 1 -- 172.21.15.178:0/1846454921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33a40f5410 msgr2=0x7f33a4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:51.172 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:51.170+0000 7f33a8d4f700 1 --2- 172.21.15.178:0/1846454921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33a40f5410 0x7f33a4105630 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f3398013510 tx=0x7f339800b040).stop 2023-11-15T04:58:51.172 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:51.170+0000 7f33a8d4f700 1 -- 172.21.15.178:0/1846454921 shutdown_connections 2023-11-15T04:58:51.172 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:51.170+0000 7f33a8d4f700 1 --2- 172.21.15.178:0/1846454921 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f338c02ff60 0x7f338c032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:51.172 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:51.170+0000 7f33a8d4f700 1 --2- 172.21.15.178:0/1846454921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33a40f5410 0x7f33a4105630 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:51.172 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:51.170+0000 7f33a8d4f700 1 -- 172.21.15.178:0/1846454921 >> 172.21.15.178:0/1846454921 conn(0x7f33a40f0b70 msgr2=0x7f33a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:51.173 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:51.170+0000 7f33a8d4f700 1 -- 172.21.15.178:0/1846454921 shutdown_connections 2023-11-15T04:58:51.173 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:51.170+0000 7f33a8d4f700 1 -- 172.21.15.178:0/1846454921 wait complete. 2023-11-15T04:58:51.173 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:58:52.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:51 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:50.340262+0000 mgr.smithi163.qesyrx (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:52.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:51 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:51.169225+0000 mon.smithi163 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.178:0/1846454921' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:58:52.557 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:58:52.557 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:53.684 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.684+0000 7ff213fff700 1 -- 172.21.15.178:0/391420808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2140f51a0 msgr2=0x7ff2140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:53.685 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.684+0000 7ff213fff700 1 --2- 172.21.15.178:0/391420808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2140f51a0 0x7ff2140f55c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7ff208009fb0 tx=0x7ff208009b30).stop 2023-11-15T04:58:53.685 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.684+0000 7ff213fff700 1 -- 172.21.15.178:0/391420808 shutdown_connections 2023-11-15T04:58:53.685 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.684+0000 7ff213fff700 1 --2- 172.21.15.178:0/391420808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2140f51a0 0x7ff2140f55c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:53.685 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.684+0000 7ff213fff700 1 -- 172.21.15.178:0/391420808 >> 172.21.15.178:0/391420808 conn(0x7ff2140f09a0 msgr2=0x7ff2140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:53.685 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.684+0000 7ff213fff700 1 -- 172.21.15.178:0/391420808 shutdown_connections 2023-11-15T04:58:53.686 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.684+0000 7ff213fff700 1 -- 172.21.15.178:0/391420808 wait complete. 2023-11-15T04:58:53.687 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.684+0000 7ff213fff700 1 Processor -- start 2023-11-15T04:58:53.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.684+0000 7ff213fff700 1 -- start start 2023-11-15T04:58:53.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.685+0000 7ff213fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2140f51a0 0x7ff2141064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:53.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.685+0000 7ff213fff700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff214106990 con 0x7ff2140f51a0 2023-11-15T04:58:53.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.685+0000 7ff212ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2140f51a0 0x7ff2141064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:53.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.685+0000 7ff212ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2140f51a0 0x7ff2141064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:49580/0 (socket says 172.21.15.178:49580) 2023-11-15T04:58:53.689 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.685+0000 7ff212ffd700 1 -- 172.21.15.178:0/3174549128 learned_addr learned my addr 172.21.15.178:0/3174549128 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:53.689 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.685+0000 7ff212ffd700 1 -- 172.21.15.178:0/3174549128 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff208005040 con 0x7ff2140f51a0 2023-11-15T04:58:53.689 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.685+0000 7ff212ffd700 1 --2- 172.21.15.178:0/3174549128 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2140f51a0 0x7ff2141064a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7ff208013510 tx=0x7ff20800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:53.689 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.686+0000 7ff1fb7fe700 1 -- 172.21.15.178:0/3174549128 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff208009320 con 0x7ff2140f51a0 2023-11-15T04:58:53.689 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.686+0000 7ff213fff700 1 -- 172.21.15.178:0/3174549128 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff214106b90 con 0x7ff2140f51a0 2023-11-15T04:58:53.690 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.686+0000 7ff1fb7fe700 1 -- 172.21.15.178:0/3174549128 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff208013f00 con 0x7ff2140f51a0 2023-11-15T04:58:53.690 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.686+0000 7ff1fb7fe700 1 -- 172.21.15.178:0/3174549128 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff208014dc0 con 0x7ff2140f51a0 2023-11-15T04:58:53.690 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.686+0000 7ff213fff700 1 -- 172.21.15.178:0/3174549128 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff214107020 con 0x7ff2140f51a0 2023-11-15T04:58:53.690 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.686+0000 7ff1fb7fe700 1 -- 172.21.15.178:0/3174549128 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff208014420 con 0x7ff2140f51a0 2023-11-15T04:58:53.690 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.687+0000 7ff1fb7fe700 1 --2- 172.21.15.178:0/3174549128 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff1fc02ff60 0x7ff1fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:53.691 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.687+0000 7ff213fff700 1 -- 172.21.15.178:0/3174549128 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff214045ad0 con 0x7ff2140f51a0 2023-11-15T04:58:53.691 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.687+0000 7ff1fb7fe700 1 -- 172.21.15.178:0/3174549128 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff208041050 con 0x7ff2140f51a0 2023-11-15T04:58:53.691 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.687+0000 7ff2127fc700 1 --2- 172.21.15.178:0/3174549128 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff1fc02ff60 0x7ff1fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:53.691 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.687+0000 7ff2127fc700 1 --2- 172.21.15.178:0/3174549128 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff1fc02ff60 0x7ff1fc032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff2040096a0 tx=0x7ff204006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:53.692 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.691+0000 7ff1fb7fe700 1 -- 172.21.15.178:0/3174549128 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff208012330 con 0x7ff2140f51a0 2023-11-15T04:58:53.899 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.899+0000 7ff213fff700 1 -- 172.21.15.178:0/3174549128 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff214107ed0 con 0x7ff2140f51a0 2023-11-15T04:58:53.900 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.900+0000 7ff1fb7fe700 1 -- 172.21.15.178:0/3174549128 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff20801cb10 con 0x7ff2140f51a0 2023-11-15T04:58:53.901 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:53.901 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:53.902 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.902+0000 7ff213fff700 1 -- 172.21.15.178:0/3174549128 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff1fc02ff60 msgr2=0x7ff1fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:53.902 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.902+0000 7ff213fff700 1 --2- 172.21.15.178:0/3174549128 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff1fc02ff60 0x7ff1fc032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff2040096a0 tx=0x7ff204006b40).stop 2023-11-15T04:58:53.902 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.902+0000 7ff213fff700 1 -- 172.21.15.178:0/3174549128 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2140f51a0 msgr2=0x7ff2141064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:53.903 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.902+0000 7ff213fff700 1 --2- 172.21.15.178:0/3174549128 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2140f51a0 0x7ff2141064a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7ff208013510 tx=0x7ff20800b040).stop 2023-11-15T04:58:53.903 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.902+0000 7ff213fff700 1 -- 172.21.15.178:0/3174549128 shutdown_connections 2023-11-15T04:58:53.903 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.902+0000 7ff213fff700 1 --2- 172.21.15.178:0/3174549128 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff1fc02ff60 0x7ff1fc032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:53.903 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.902+0000 7ff213fff700 1 --2- 172.21.15.178:0/3174549128 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2140f51a0 0x7ff2141064a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:53.903 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.902+0000 7ff213fff700 1 -- 172.21.15.178:0/3174549128 >> 172.21.15.178:0/3174549128 conn(0x7ff2140f09a0 msgr2=0x7ff21405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:53.904 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.902+0000 7ff213fff700 1 -- 172.21.15.178:0/3174549128 shutdown_connections 2023-11-15T04:58:53.904 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:53.902+0000 7ff213fff700 1 -- 172.21.15.178:0/3174549128 wait complete. 2023-11-15T04:58:53.904 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:58:54.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:53 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:52.340642+0000 mgr.smithi163.qesyrx (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:54.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:53 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:53.900669+0000 mon.smithi163 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.178:0/3174549128' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:58:55.280 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:58:55.281 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:56.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:55 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:54.340926+0000 mgr.smithi163.qesyrx (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:56.524 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.522+0000 7fdd92161700 1 -- 172.21.15.178:0/2993097619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0601e0 msgr2=0x7fdd8c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:56.524 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.522+0000 7fdd92161700 1 --2- 172.21.15.178:0/2993097619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0601e0 0x7fdd8c0f8f20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fdd88009fb0 tx=0x7fdd88009b30).stop 2023-11-15T04:58:56.524 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.522+0000 7fdd92161700 1 -- 172.21.15.178:0/2993097619 shutdown_connections 2023-11-15T04:58:56.525 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.522+0000 7fdd92161700 1 --2- 172.21.15.178:0/2993097619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0601e0 0x7fdd8c0f8f20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:56.525 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.522+0000 7fdd92161700 1 -- 172.21.15.178:0/2993097619 >> 172.21.15.178:0/2993097619 conn(0x7fdd8c0f09a0 msgr2=0x7fdd8c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:56.525 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.522+0000 7fdd92161700 1 -- 172.21.15.178:0/2993097619 shutdown_connections 2023-11-15T04:58:56.525 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.522+0000 7fdd92161700 1 -- 172.21.15.178:0/2993097619 wait complete. 2023-11-15T04:58:56.525 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.522+0000 7fdd92161700 1 Processor -- start 2023-11-15T04:58:56.526 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.522+0000 7fdd92161700 1 -- start start 2023-11-15T04:58:56.526 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.522+0000 7fdd92161700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0601e0 0x7fdd8c0f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:56.526 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.522+0000 7fdd92161700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c0f4e20 con 0x7fdd8c0601e0 2023-11-15T04:58:56.526 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.523+0000 7fdd9115f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0601e0 0x7fdd8c0f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:56.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.523+0000 7fdd9115f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0601e0 0x7fdd8c0f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:52862/0 (socket says 172.21.15.178:52862) 2023-11-15T04:58:56.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.523+0000 7fdd9115f700 1 -- 172.21.15.178:0/1596139820 learned_addr learned my addr 172.21.15.178:0/1596139820 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:56.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.523+0000 7fdd9115f700 1 -- 172.21.15.178:0/1596139820 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd88005040 con 0x7fdd8c0601e0 2023-11-15T04:58:56.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.523+0000 7fdd9115f700 1 --2- 172.21.15.178:0/1596139820 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0601e0 0x7fdd8c0f69a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fdd88013510 tx=0x7fdd8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:56.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.523+0000 7fdd7dffb700 1 -- 172.21.15.178:0/1596139820 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd88009320 con 0x7fdd8c0601e0 2023-11-15T04:58:56.528 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.523+0000 7fdd92161700 1 -- 172.21.15.178:0/1596139820 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd8c0f4fc0 con 0x7fdd8c0601e0 2023-11-15T04:58:56.528 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.523+0000 7fdd7dffb700 1 -- 172.21.15.178:0/1596139820 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd88013f00 con 0x7fdd8c0601e0 2023-11-15T04:58:56.528 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.523+0000 7fdd7dffb700 1 -- 172.21.15.178:0/1596139820 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd88014d90 con 0x7fdd8c0601e0 2023-11-15T04:58:56.529 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.523+0000 7fdd92161700 1 -- 172.21.15.178:0/1596139820 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd8c0f5450 con 0x7fdd8c0601e0 2023-11-15T04:58:56.530 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.524+0000 7fdd7dffb700 1 -- 172.21.15.178:0/1596139820 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd880095f0 con 0x7fdd8c0601e0 2023-11-15T04:58:56.530 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.524+0000 7fdd92161700 1 -- 172.21.15.178:0/1596139820 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd8c059100 con 0x7fdd8c0601e0 2023-11-15T04:58:56.530 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.524+0000 7fdd7dffb700 1 --2- 172.21.15.178:0/1596139820 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdd7802ffb0 0x7fdd78032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:56.531 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.524+0000 7fdd7dffb700 1 -- 172.21.15.178:0/1596139820 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd880410c0 con 0x7fdd8c0601e0 2023-11-15T04:58:56.531 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.525+0000 7fdd9095e700 1 --2- 172.21.15.178:0/1596139820 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdd7802ffb0 0x7fdd78032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:56.531 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.525+0000 7fdd9095e700 1 --2- 172.21.15.178:0/1596139820 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdd7802ffb0 0x7fdd78032470 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fdd800096a0 tx=0x7fdd80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:56.532 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.529+0000 7fdd7dffb700 1 -- 172.21.15.178:0/1596139820 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd88014420 con 0x7fdd8c0601e0 2023-11-15T04:58:56.736 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.736+0000 7fdd92161700 1 -- 172.21.15.178:0/1596139820 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd8c0fbcd0 con 0x7fdd8c0601e0 2023-11-15T04:58:56.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.736+0000 7fdd7dffb700 1 -- 172.21.15.178:0/1596139820 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdd8801d7f0 con 0x7fdd8c0601e0 2023-11-15T04:58:56.737 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:56.737 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:56.738 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.738+0000 7fdd92161700 1 -- 172.21.15.178:0/1596139820 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdd7802ffb0 msgr2=0x7fdd78032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:56.739 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.738+0000 7fdd92161700 1 --2- 172.21.15.178:0/1596139820 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdd7802ffb0 0x7fdd78032470 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fdd800096a0 tx=0x7fdd80006b40).stop 2023-11-15T04:58:56.739 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.738+0000 7fdd92161700 1 -- 172.21.15.178:0/1596139820 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0601e0 msgr2=0x7fdd8c0f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:56.739 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.738+0000 7fdd92161700 1 --2- 172.21.15.178:0/1596139820 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0601e0 0x7fdd8c0f69a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fdd88013510 tx=0x7fdd8800b040).stop 2023-11-15T04:58:56.739 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.738+0000 7fdd92161700 1 -- 172.21.15.178:0/1596139820 shutdown_connections 2023-11-15T04:58:56.739 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.738+0000 7fdd92161700 1 --2- 172.21.15.178:0/1596139820 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdd7802ffb0 0x7fdd78032470 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:56.740 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.738+0000 7fdd92161700 1 --2- 172.21.15.178:0/1596139820 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0601e0 0x7fdd8c0f69a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:56.740 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.738+0000 7fdd92161700 1 -- 172.21.15.178:0/1596139820 >> 172.21.15.178:0/1596139820 conn(0x7fdd8c0f09a0 msgr2=0x7fdd8c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:56.740 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.738+0000 7fdd92161700 1 -- 172.21.15.178:0/1596139820 shutdown_connections 2023-11-15T04:58:56.740 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:56.738+0000 7fdd92161700 1 -- 172.21.15.178:0/1596139820 wait complete. 2023-11-15T04:58:56.741 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:58:57.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:56 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:56.736855+0000 mon.smithi163 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.178:0/1596139820' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:58:58.110 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:58:58.111 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:58:58.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:57 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:56.341278+0000 mgr.smithi163.qesyrx (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:58:59.382 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.381+0000 7fc8386e0700 1 -- 172.21.15.178:0/2659968064 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8300f6510 msgr2=0x7fc8300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:59.382 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.381+0000 7fc8386e0700 1 --2- 172.21.15.178:0/2659968064 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8300f6510 0x7fc8300f6930 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fc828009fb0 tx=0x7fc828009b30).stop 2023-11-15T04:58:59.383 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.381+0000 7fc8386e0700 1 -- 172.21.15.178:0/2659968064 shutdown_connections 2023-11-15T04:58:59.383 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.381+0000 7fc8386e0700 1 --2- 172.21.15.178:0/2659968064 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8300f6510 0x7fc8300f6930 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:59.383 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.381+0000 7fc8386e0700 1 -- 172.21.15.178:0/2659968064 >> 172.21.15.178:0/2659968064 conn(0x7fc8300f1c70 msgr2=0x7fc8300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:59.383 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.381+0000 7fc8386e0700 1 -- 172.21.15.178:0/2659968064 shutdown_connections 2023-11-15T04:58:59.383 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.381+0000 7fc8386e0700 1 -- 172.21.15.178:0/2659968064 wait complete. 2023-11-15T04:58:59.384 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.382+0000 7fc8386e0700 1 Processor -- start 2023-11-15T04:58:59.384 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.382+0000 7fc8386e0700 1 -- start start 2023-11-15T04:58:59.384 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.382+0000 7fc8386e0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8300f6510 0x7fc830106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:59.384 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.382+0000 7fc8386e0700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc830106c20 con 0x7fc8300f6510 2023-11-15T04:58:59.386 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.382+0000 7fc83647c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8300f6510 0x7fc830106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:59.386 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.382+0000 7fc83647c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8300f6510 0x7fc830106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:52886/0 (socket says 172.21.15.178:52886) 2023-11-15T04:58:59.387 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.382+0000 7fc83647c700 1 -- 172.21.15.178:0/2013338096 learned_addr learned my addr 172.21.15.178:0/2013338096 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:58:59.387 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.383+0000 7fc83647c700 1 -- 172.21.15.178:0/2013338096 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc828005040 con 0x7fc8300f6510 2023-11-15T04:58:59.387 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.383+0000 7fc83647c700 1 --2- 172.21.15.178:0/2013338096 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8300f6510 0x7fc830106730 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fc828013510 tx=0x7fc82800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:59.387 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.383+0000 7fc82effd700 1 -- 172.21.15.178:0/2013338096 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc828009320 con 0x7fc8300f6510 2023-11-15T04:58:59.387 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.383+0000 7fc8386e0700 1 -- 172.21.15.178:0/2013338096 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc830106e20 con 0x7fc8300f6510 2023-11-15T04:58:59.388 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.383+0000 7fc82effd700 1 -- 172.21.15.178:0/2013338096 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc828013f00 con 0x7fc8300f6510 2023-11-15T04:58:59.388 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.383+0000 7fc82effd700 1 -- 172.21.15.178:0/2013338096 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc828014d90 con 0x7fc8300f6510 2023-11-15T04:58:59.388 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.383+0000 7fc8386e0700 1 -- 172.21.15.178:0/2013338096 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8301072b0 con 0x7fc8300f6510 2023-11-15T04:58:59.388 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.384+0000 7fc82effd700 1 -- 172.21.15.178:0/2013338096 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc8280095f0 con 0x7fc8300f6510 2023-11-15T04:58:59.388 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.384+0000 7fc8386e0700 1 -- 172.21.15.178:0/2013338096 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc830045ad0 con 0x7fc8300f6510 2023-11-15T04:58:59.389 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.384+0000 7fc82effd700 1 --2- 172.21.15.178:0/2013338096 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc81802ff60 0x7fc818032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:58:59.389 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.384+0000 7fc82effd700 1 -- 172.21.15.178:0/2013338096 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc828040fc0 con 0x7fc8300f6510 2023-11-15T04:58:59.389 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.384+0000 7fc835c7b700 1 --2- 172.21.15.178:0/2013338096 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc81802ff60 0x7fc818032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:58:59.389 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.384+0000 7fc835c7b700 1 --2- 172.21.15.178:0/2013338096 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc81802ff60 0x7fc818032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fc8240096a0 tx=0x7fc824006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:58:59.390 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.389+0000 7fc82effd700 1 -- 172.21.15.178:0/2013338096 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc828014420 con 0x7fc8300f6510 2023-11-15T04:58:59.595 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.595+0000 7fc8386e0700 1 -- 172.21.15.178:0/2013338096 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8300fbf60 con 0x7fc8300f6510 2023-11-15T04:58:59.596 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.595+0000 7fc82effd700 1 -- 172.21.15.178:0/2013338096 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc828010230 con 0x7fc8300f6510 2023-11-15T04:58:59.596 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:58:59.596 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:58:59.597 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.597+0000 7fc8386e0700 1 -- 172.21.15.178:0/2013338096 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc81802ff60 msgr2=0x7fc818032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:59.597 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.597+0000 7fc8386e0700 1 --2- 172.21.15.178:0/2013338096 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc81802ff60 0x7fc818032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fc8240096a0 tx=0x7fc824006b40).stop 2023-11-15T04:58:59.597 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.597+0000 7fc8386e0700 1 -- 172.21.15.178:0/2013338096 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8300f6510 msgr2=0x7fc830106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:58:59.598 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.597+0000 7fc8386e0700 1 --2- 172.21.15.178:0/2013338096 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8300f6510 0x7fc830106730 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fc828013510 tx=0x7fc82800b040).stop 2023-11-15T04:58:59.598 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.597+0000 7fc8386e0700 1 -- 172.21.15.178:0/2013338096 shutdown_connections 2023-11-15T04:58:59.598 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.597+0000 7fc8386e0700 1 --2- 172.21.15.178:0/2013338096 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc81802ff60 0x7fc818032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:59.598 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.597+0000 7fc8386e0700 1 --2- 172.21.15.178:0/2013338096 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8300f6510 0x7fc830106730 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:58:59.599 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.597+0000 7fc8386e0700 1 -- 172.21.15.178:0/2013338096 >> 172.21.15.178:0/2013338096 conn(0x7fc8300f1c70 msgr2=0x7fc8300f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:58:59.599 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.597+0000 7fc8386e0700 1 -- 172.21.15.178:0/2013338096 shutdown_connections 2023-11-15T04:58:59.599 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:58:59.597+0000 7fc8386e0700 1 -- 172.21.15.178:0/2013338096 wait complete. 2023-11-15T04:58:59.599 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:00.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:59 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:58:58.341664+0000 mgr.smithi163.qesyrx (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:00.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:58:59 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:58:59.596038+0000 mon.smithi163 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.178:0/2013338096' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:59:00.978 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:59:00.979 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:59:02.183 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.182+0000 7fc48e37c700 1 -- 172.21.15.178:0/383181678 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4880f62a0 msgr2=0x7fc4880f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:02.183 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.182+0000 7fc48e37c700 1 --2- 172.21.15.178:0/383181678 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4880f62a0 0x7fc4880f66c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fc484009fb0 tx=0x7fc484009b30).stop 2023-11-15T04:59:02.184 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.182+0000 7fc48e37c700 1 -- 172.21.15.178:0/383181678 shutdown_connections 2023-11-15T04:59:02.184 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.182+0000 7fc48e37c700 1 --2- 172.21.15.178:0/383181678 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4880f62a0 0x7fc4880f66c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:02.184 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.182+0000 7fc48e37c700 1 -- 172.21.15.178:0/383181678 >> 172.21.15.178:0/383181678 conn(0x7fc4880f1aa0 msgr2=0x7fc4880f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:02.184 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.182+0000 7fc48e37c700 1 -- 172.21.15.178:0/383181678 shutdown_connections 2023-11-15T04:59:02.185 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.182+0000 7fc48e37c700 1 -- 172.21.15.178:0/383181678 wait complete. 2023-11-15T04:59:02.185 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.183+0000 7fc48e37c700 1 Processor -- start 2023-11-15T04:59:02.185 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.183+0000 7fc48e37c700 1 -- start start 2023-11-15T04:59:02.185 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.183+0000 7fc48e37c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4880f62a0 0x7fc4881064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:02.185 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.183+0000 7fc48e37c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4881069a0 con 0x7fc4880f62a0 2023-11-15T04:59:02.186 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.183+0000 7fc48d37a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4880f62a0 0x7fc4881064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:02.186 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.183+0000 7fc48d37a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4880f62a0 0x7fc4881064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:52908/0 (socket says 172.21.15.178:52908) 2023-11-15T04:59:02.186 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.183+0000 7fc48d37a700 1 -- 172.21.15.178:0/3095714269 learned_addr learned my addr 172.21.15.178:0/3095714269 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:59:02.187 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.183+0000 7fc48d37a700 1 -- 172.21.15.178:0/3095714269 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc484005040 con 0x7fc4880f62a0 2023-11-15T04:59:02.188 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.184+0000 7fc48d37a700 1 --2- 172.21.15.178:0/3095714269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4880f62a0 0x7fc4881064b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fc484013510 tx=0x7fc48400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:02.188 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.184+0000 7fc47dffb700 1 -- 172.21.15.178:0/3095714269 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc484009320 con 0x7fc4880f62a0 2023-11-15T04:59:02.188 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.184+0000 7fc47dffb700 1 -- 172.21.15.178:0/3095714269 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc484013f00 con 0x7fc4880f62a0 2023-11-15T04:59:02.188 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.184+0000 7fc47dffb700 1 -- 172.21.15.178:0/3095714269 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc484014dc0 con 0x7fc4880f62a0 2023-11-15T04:59:02.188 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.184+0000 7fc48e37c700 1 -- 172.21.15.178:0/3095714269 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc488106ba0 con 0x7fc4880f62a0 2023-11-15T04:59:02.189 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.184+0000 7fc48e37c700 1 -- 172.21.15.178:0/3095714269 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc488107030 con 0x7fc4880f62a0 2023-11-15T04:59:02.189 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.185+0000 7fc47dffb700 1 -- 172.21.15.178:0/3095714269 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc4840095f0 con 0x7fc4880f62a0 2023-11-15T04:59:02.189 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.185+0000 7fc47dffb700 1 --2- 172.21.15.178:0/3095714269 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc47402ff60 0x7fc474032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:02.189 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.185+0000 7fc48e37c700 1 -- 172.21.15.178:0/3095714269 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc488045ad0 con 0x7fc4880f62a0 2023-11-15T04:59:02.189 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.185+0000 7fc47dffb700 1 -- 172.21.15.178:0/3095714269 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc484041040 con 0x7fc4880f62a0 2023-11-15T04:59:02.190 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.185+0000 7fc48cb79700 1 --2- 172.21.15.178:0/3095714269 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc47402ff60 0x7fc474032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:02.190 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.186+0000 7fc48cb79700 1 --2- 172.21.15.178:0/3095714269 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc47402ff60 0x7fc474032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fc4780096a0 tx=0x7fc478006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:02.190 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.190+0000 7fc47dffb700 1 -- 172.21.15.178:0/3095714269 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc484014420 con 0x7fc4880f62a0 2023-11-15T04:59:02.395 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.394+0000 7fc48e37c700 1 -- 172.21.15.178:0/3095714269 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc488107ee0 con 0x7fc4880f62a0 2023-11-15T04:59:02.395 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.395+0000 7fc47dffb700 1 -- 172.21.15.178:0/3095714269 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc484010230 con 0x7fc4880f62a0 2023-11-15T04:59:02.396 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:59:02.396 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:59:02.397 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.396+0000 7fc48e37c700 1 -- 172.21.15.178:0/3095714269 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc47402ff60 msgr2=0x7fc474032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:02.397 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.396+0000 7fc48e37c700 1 --2- 172.21.15.178:0/3095714269 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc47402ff60 0x7fc474032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fc4780096a0 tx=0x7fc478006b40).stop 2023-11-15T04:59:02.397 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.397+0000 7fc48e37c700 1 -- 172.21.15.178:0/3095714269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4880f62a0 msgr2=0x7fc4881064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:02.398 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.397+0000 7fc48e37c700 1 --2- 172.21.15.178:0/3095714269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4880f62a0 0x7fc4881064b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fc484013510 tx=0x7fc48400b040).stop 2023-11-15T04:59:02.398 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.397+0000 7fc48e37c700 1 -- 172.21.15.178:0/3095714269 shutdown_connections 2023-11-15T04:59:02.398 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.397+0000 7fc48e37c700 1 --2- 172.21.15.178:0/3095714269 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc47402ff60 0x7fc474032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:02.398 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.397+0000 7fc48e37c700 1 --2- 172.21.15.178:0/3095714269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4880f62a0 0x7fc4881064b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:02.398 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.397+0000 7fc48e37c700 1 -- 172.21.15.178:0/3095714269 >> 172.21.15.178:0/3095714269 conn(0x7fc4880f1aa0 msgr2=0x7fc4880f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:02.399 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.397+0000 7fc48e37c700 1 -- 172.21.15.178:0/3095714269 shutdown_connections 2023-11-15T04:59:02.399 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:02.397+0000 7fc48e37c700 1 -- 172.21.15.178:0/3095714269 wait complete. 2023-11-15T04:59:02.399 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:02.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:01 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:00.342001+0000 mgr.smithi163.qesyrx (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:03.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:02 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:02.395745+0000 mon.smithi163 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.178:0/3095714269' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:59:03.770 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:59:03.771 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:59:04.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:03 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:02.342249+0000 mgr.smithi163.qesyrx (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:04.927 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.923+0000 7fdcb9d9f700 1 -- 172.21.15.178:0/1682173173 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdcb40f62a0 msgr2=0x7fdcb40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:04.928 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.923+0000 7fdcb9d9f700 1 --2- 172.21.15.178:0/1682173173 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdcb40f62a0 0x7fdcb40f66c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fdca8009fb0 tx=0x7fdca8009b30).stop 2023-11-15T04:59:04.928 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.923+0000 7fdcb9d9f700 1 -- 172.21.15.178:0/1682173173 shutdown_connections 2023-11-15T04:59:04.928 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.923+0000 7fdcb9d9f700 1 --2- 172.21.15.178:0/1682173173 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdcb40f62a0 0x7fdcb40f66c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:04.928 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.923+0000 7fdcb9d9f700 1 -- 172.21.15.178:0/1682173173 >> 172.21.15.178:0/1682173173 conn(0x7fdcb40f1aa0 msgr2=0x7fdcb40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:04.928 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.923+0000 7fdcb9d9f700 1 -- 172.21.15.178:0/1682173173 shutdown_connections 2023-11-15T04:59:04.929 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.923+0000 7fdcb9d9f700 1 -- 172.21.15.178:0/1682173173 wait complete. 2023-11-15T04:59:04.929 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.924+0000 7fdcb9d9f700 1 Processor -- start 2023-11-15T04:59:04.929 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.924+0000 7fdcb9d9f700 1 -- start start 2023-11-15T04:59:04.929 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.924+0000 7fdcb9d9f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdcb40f62a0 0x7fdcb41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:04.929 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.924+0000 7fdcb9d9f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb41069a0 con 0x7fdcb40f62a0 2023-11-15T04:59:04.929 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.924+0000 7fdcb8d9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdcb40f62a0 0x7fdcb41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:04.930 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.925+0000 7fdcb8d9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdcb40f62a0 0x7fdcb41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:57384/0 (socket says 172.21.15.178:57384) 2023-11-15T04:59:04.930 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.925+0000 7fdcb8d9d700 1 -- 172.21.15.178:0/3900803443 learned_addr learned my addr 172.21.15.178:0/3900803443 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:59:04.931 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.925+0000 7fdcb8d9d700 1 -- 172.21.15.178:0/3900803443 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca8005040 con 0x7fdcb40f62a0 2023-11-15T04:59:04.931 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.925+0000 7fdcb8d9d700 1 --2- 172.21.15.178:0/3900803443 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdcb40f62a0 0x7fdcb41064b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fdca8013510 tx=0x7fdca800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:04.931 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.925+0000 7fdcb17fa700 1 -- 172.21.15.178:0/3900803443 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdca8009320 con 0x7fdcb40f62a0 2023-11-15T04:59:04.931 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.925+0000 7fdcb9d9f700 1 -- 172.21.15.178:0/3900803443 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb4106ba0 con 0x7fdcb40f62a0 2023-11-15T04:59:04.931 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.925+0000 7fdcb17fa700 1 -- 172.21.15.178:0/3900803443 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdca8013f00 con 0x7fdcb40f62a0 2023-11-15T04:59:04.932 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.925+0000 7fdcb17fa700 1 -- 172.21.15.178:0/3900803443 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdca8014dc0 con 0x7fdcb40f62a0 2023-11-15T04:59:04.932 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.925+0000 7fdcb9d9f700 1 -- 172.21.15.178:0/3900803443 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb4107030 con 0x7fdcb40f62a0 2023-11-15T04:59:04.932 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.926+0000 7fdcb17fa700 1 -- 172.21.15.178:0/3900803443 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdca80095f0 con 0x7fdcb40f62a0 2023-11-15T04:59:04.932 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.926+0000 7fdcb9d9f700 1 -- 172.21.15.178:0/3900803443 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb4045ad0 con 0x7fdcb40f62a0 2023-11-15T04:59:04.932 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.926+0000 7fdcb17fa700 1 --2- 172.21.15.178:0/3900803443 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdca402ff60 0x7fdca4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:04.933 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.926+0000 7fdcb17fa700 1 -- 172.21.15.178:0/3900803443 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdca8041020 con 0x7fdcb40f62a0 2023-11-15T04:59:04.933 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.926+0000 7fdcb3fff700 1 --2- 172.21.15.178:0/3900803443 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdca402ff60 0x7fdca4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:04.933 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.927+0000 7fdcb3fff700 1 --2- 172.21.15.178:0/3900803443 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdca402ff60 0x7fdca4032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fdca00096a0 tx=0x7fdca0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:04.934 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:04.931+0000 7fdcb17fa700 1 -- 172.21.15.178:0/3900803443 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdca8014420 con 0x7fdcb40f62a0 2023-11-15T04:59:05.137 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:05.137+0000 7fdcb9d9f700 1 -- 172.21.15.178:0/3900803443 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdcb4107e80 con 0x7fdcb40f62a0 2023-11-15T04:59:05.137 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:05.137+0000 7fdcb17fa700 1 -- 172.21.15.178:0/3900803443 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdca801c920 con 0x7fdcb40f62a0 2023-11-15T04:59:05.138 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:59:05.138 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:59:05.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:05.139+0000 7fdcb9d9f700 1 -- 172.21.15.178:0/3900803443 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdca402ff60 msgr2=0x7fdca4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:05.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:05.139+0000 7fdcb9d9f700 1 --2- 172.21.15.178:0/3900803443 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdca402ff60 0x7fdca4032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fdca00096a0 tx=0x7fdca0006b40).stop 2023-11-15T04:59:05.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:05.139+0000 7fdcb9d9f700 1 -- 172.21.15.178:0/3900803443 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdcb40f62a0 msgr2=0x7fdcb41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:05.140 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:05.139+0000 7fdcb9d9f700 1 --2- 172.21.15.178:0/3900803443 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdcb40f62a0 0x7fdcb41064b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fdca8013510 tx=0x7fdca800b040).stop 2023-11-15T04:59:05.140 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:05.139+0000 7fdcb9d9f700 1 -- 172.21.15.178:0/3900803443 shutdown_connections 2023-11-15T04:59:05.140 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:05.139+0000 7fdcb9d9f700 1 --2- 172.21.15.178:0/3900803443 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdca402ff60 0x7fdca4032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:05.140 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:05.139+0000 7fdcb9d9f700 1 --2- 172.21.15.178:0/3900803443 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdcb40f62a0 0x7fdcb41064b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:05.140 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:05.139+0000 7fdcb9d9f700 1 -- 172.21.15.178:0/3900803443 >> 172.21.15.178:0/3900803443 conn(0x7fdcb40f1aa0 msgr2=0x7fdcb40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:05.140 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:05.139+0000 7fdcb9d9f700 1 -- 172.21.15.178:0/3900803443 shutdown_connections 2023-11-15T04:59:05.141 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:05.139+0000 7fdcb9d9f700 1 -- 172.21.15.178:0/3900803443 wait complete. 2023-11-15T04:59:05.141 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:06.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:05 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:04.342623+0000 mgr.smithi163.qesyrx (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:06.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:05 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:05.137881+0000 mon.smithi163 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.178:0/3900803443' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:59:06.519 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:59:06.520 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:59:07.754 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.753+0000 7f5389528700 1 -- 172.21.15.178:0/2423483548 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53840f51a0 msgr2=0x7f53840f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:07.755 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.753+0000 7f5389528700 1 --2- 172.21.15.178:0/2423483548 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53840f51a0 0x7f53840f55c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f5374009fb0 tx=0x7f5374009b30).stop 2023-11-15T04:59:07.755 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.753+0000 7f5389528700 1 -- 172.21.15.178:0/2423483548 shutdown_connections 2023-11-15T04:59:07.755 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.753+0000 7f5389528700 1 --2- 172.21.15.178:0/2423483548 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53840f51a0 0x7f53840f55c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:07.755 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.753+0000 7f5389528700 1 -- 172.21.15.178:0/2423483548 >> 172.21.15.178:0/2423483548 conn(0x7f53840f09a0 msgr2=0x7f53840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:07.755 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.753+0000 7f5389528700 1 -- 172.21.15.178:0/2423483548 shutdown_connections 2023-11-15T04:59:07.756 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.753+0000 7f5389528700 1 -- 172.21.15.178:0/2423483548 wait complete. 2023-11-15T04:59:07.756 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.754+0000 7f5389528700 1 Processor -- start 2023-11-15T04:59:07.756 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.754+0000 7f5389528700 1 -- start start 2023-11-15T04:59:07.756 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.754+0000 7f5389528700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53840601e0 0x7f5384109360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:07.756 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.754+0000 7f5389528700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5384109850 con 0x7f53840601e0 2023-11-15T04:59:07.758 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.754+0000 7f5383fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53840601e0 0x7f5384109360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:07.758 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.754+0000 7f5383fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53840601e0 0x7f5384109360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:57404/0 (socket says 172.21.15.178:57404) 2023-11-15T04:59:07.759 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.754+0000 7f5383fff700 1 -- 172.21.15.178:0/1164854760 learned_addr learned my addr 172.21.15.178:0/1164854760 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:59:07.759 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.755+0000 7f5383fff700 1 -- 172.21.15.178:0/1164854760 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5374005040 con 0x7f53840601e0 2023-11-15T04:59:07.759 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.755+0000 7f5383fff700 1 --2- 172.21.15.178:0/1164854760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53840601e0 0x7f5384109360 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f5374009f80 tx=0x7f5374009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:07.759 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.755+0000 7f5380ff9700 1 -- 172.21.15.178:0/1164854760 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53740136c0 con 0x7f53840601e0 2023-11-15T04:59:07.759 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.755+0000 7f5389528700 1 -- 172.21.15.178:0/1164854760 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5384109990 con 0x7f53840601e0 2023-11-15T04:59:07.760 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.755+0000 7f5380ff9700 1 -- 172.21.15.178:0/1164854760 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53740095f0 con 0x7f53840601e0 2023-11-15T04:59:07.760 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.755+0000 7f5380ff9700 1 -- 172.21.15.178:0/1164854760 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5374014420 con 0x7f53840601e0 2023-11-15T04:59:07.760 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.755+0000 7f5389528700 1 -- 172.21.15.178:0/1164854760 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5384109dc0 con 0x7f53840601e0 2023-11-15T04:59:07.760 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.756+0000 7f5380ff9700 1 -- 172.21.15.178:0/1164854760 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5374025060 con 0x7f53840601e0 2023-11-15T04:59:07.760 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.756+0000 7f5389528700 1 -- 172.21.15.178:0/1164854760 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5384045ad0 con 0x7f53840601e0 2023-11-15T04:59:07.761 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.756+0000 7f5380ff9700 1 --2- 172.21.15.178:0/1164854760 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f536c02ff60 0x7f536c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:07.761 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.756+0000 7f5380ff9700 1 -- 172.21.15.178:0/1164854760 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5374040de0 con 0x7f53840601e0 2023-11-15T04:59:07.761 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.756+0000 7f53837fe700 1 --2- 172.21.15.178:0/1164854760 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f536c02ff60 0x7f536c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:07.761 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.757+0000 7f53837fe700 1 --2- 172.21.15.178:0/1164854760 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f536c02ff60 0x7f536c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f53780096a0 tx=0x7f5378006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:07.762 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.761+0000 7f5380ff9700 1 -- 172.21.15.178:0/1164854760 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f537400f280 con 0x7f53840601e0 2023-11-15T04:59:07.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.963+0000 7f5389528700 1 -- 172.21.15.178:0/1164854760 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53840fabe0 con 0x7f53840601e0 2023-11-15T04:59:07.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.963+0000 7f5380ff9700 1 -- 172.21.15.178:0/1164854760 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5374004490 con 0x7f53840601e0 2023-11-15T04:59:07.964 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:59:07.964 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:59:07.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.965+0000 7f5389528700 1 -- 172.21.15.178:0/1164854760 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f536c02ff60 msgr2=0x7f536c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:07.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.965+0000 7f5389528700 1 --2- 172.21.15.178:0/1164854760 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f536c02ff60 0x7f536c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f53780096a0 tx=0x7f5378006b40).stop 2023-11-15T04:59:07.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.965+0000 7f5389528700 1 -- 172.21.15.178:0/1164854760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53840601e0 msgr2=0x7f5384109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:07.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.965+0000 7f5389528700 1 --2- 172.21.15.178:0/1164854760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53840601e0 0x7f5384109360 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f5374009f80 tx=0x7f5374009b30).stop 2023-11-15T04:59:07.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.965+0000 7f5389528700 1 -- 172.21.15.178:0/1164854760 shutdown_connections 2023-11-15T04:59:07.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.965+0000 7f5389528700 1 --2- 172.21.15.178:0/1164854760 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f536c02ff60 0x7f536c032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:07.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.965+0000 7f5389528700 1 --2- 172.21.15.178:0/1164854760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53840601e0 0x7f5384109360 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:07.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.965+0000 7f5389528700 1 -- 172.21.15.178:0/1164854760 >> 172.21.15.178:0/1164854760 conn(0x7f53840f09a0 msgr2=0x7f53840f1500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:07.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.965+0000 7f5389528700 1 -- 172.21.15.178:0/1164854760 shutdown_connections 2023-11-15T04:59:07.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:07.965+0000 7f5389528700 1 -- 172.21.15.178:0/1164854760 wait complete. 2023-11-15T04:59:07.968 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:08.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:07 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:06.342989+0000 mgr.smithi163.qesyrx (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:09.359 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:59:09.360 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:59:09.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:08 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:07.964254+0000 mon.smithi163 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.178:0/1164854760' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:59:10.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:09 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:08.343336+0000 mgr.smithi163.qesyrx (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:10.669 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.668+0000 7f6cf9882700 1 -- 172.21.15.178:0/3598279572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cf40f53f0 msgr2=0x7f6cf40f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:10.669 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.668+0000 7f6cf9882700 1 --2- 172.21.15.178:0/3598279572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cf40f53f0 0x7f6cf40f5810 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f6cdc009fb0 tx=0x7f6cdc009b30).stop 2023-11-15T04:59:10.669 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.668+0000 7f6cf9882700 1 -- 172.21.15.178:0/3598279572 shutdown_connections 2023-11-15T04:59:10.670 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.668+0000 7f6cf9882700 1 --2- 172.21.15.178:0/3598279572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cf40f53f0 0x7f6cf40f5810 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:10.670 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.668+0000 7f6cf9882700 1 -- 172.21.15.178:0/3598279572 >> 172.21.15.178:0/3598279572 conn(0x7f6cf40f0b70 msgr2=0x7f6cf40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:10.670 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.669+0000 7f6cf9882700 1 -- 172.21.15.178:0/3598279572 shutdown_connections 2023-11-15T04:59:10.670 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.669+0000 7f6cf9882700 1 -- 172.21.15.178:0/3598279572 wait complete. 2023-11-15T04:59:10.672 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.669+0000 7f6cf9882700 1 Processor -- start 2023-11-15T04:59:10.673 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.669+0000 7f6cf9882700 1 -- start start 2023-11-15T04:59:10.673 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.669+0000 7f6cf9882700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cf40f53f0 0x7f6cf41055e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:10.673 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.669+0000 7f6cf9882700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf4105ad0 con 0x7f6cf40f53f0 2023-11-15T04:59:10.673 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.670+0000 7f6cf2ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cf40f53f0 0x7f6cf41055e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:10.673 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.670+0000 7f6cf2ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cf40f53f0 0x7f6cf41055e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:57430/0 (socket says 172.21.15.178:57430) 2023-11-15T04:59:10.674 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.670+0000 7f6cf2ffd700 1 -- 172.21.15.178:0/128052321 learned_addr learned my addr 172.21.15.178:0/128052321 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:59:10.674 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.670+0000 7f6cf2ffd700 1 -- 172.21.15.178:0/128052321 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc005040 con 0x7f6cf40f53f0 2023-11-15T04:59:10.674 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.670+0000 7f6cf2ffd700 1 --2- 172.21.15.178:0/128052321 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cf40f53f0 0x7f6cf41055e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f6cdc013510 tx=0x7f6cdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:10.674 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.670+0000 7f6ceb7fe700 1 -- 172.21.15.178:0/128052321 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cdc009320 con 0x7f6cf40f53f0 2023-11-15T04:59:10.674 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.670+0000 7f6cf9882700 1 -- 172.21.15.178:0/128052321 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf4105cd0 con 0x7f6cf40f53f0 2023-11-15T04:59:10.675 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.670+0000 7f6ceb7fe700 1 -- 172.21.15.178:0/128052321 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6cdc013f00 con 0x7f6cf40f53f0 2023-11-15T04:59:10.675 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.670+0000 7f6ceb7fe700 1 -- 172.21.15.178:0/128052321 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cdc014d90 con 0x7f6cf40f53f0 2023-11-15T04:59:10.675 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.670+0000 7f6cf9882700 1 -- 172.21.15.178:0/128052321 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf4106160 con 0x7f6cf40f53f0 2023-11-15T04:59:10.676 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.671+0000 7f6ceb7fe700 1 -- 172.21.15.178:0/128052321 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6cdc014420 con 0x7f6cf40f53f0 2023-11-15T04:59:10.676 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.671+0000 7f6cf9882700 1 -- 172.21.15.178:0/128052321 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf4045ad0 con 0x7f6cf40f53f0 2023-11-15T04:59:10.676 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.671+0000 7f6ceb7fe700 1 --2- 172.21.15.178:0/128052321 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6ce002ff60 0x7f6ce0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:10.677 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.671+0000 7f6ceb7fe700 1 -- 172.21.15.178:0/128052321 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6cdc040fb0 con 0x7f6cf40f53f0 2023-11-15T04:59:10.677 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.671+0000 7f6cf27fc700 1 --2- 172.21.15.178:0/128052321 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6ce002ff60 0x7f6ce0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:10.677 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.672+0000 7f6cf27fc700 1 --2- 172.21.15.178:0/128052321 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6ce002ff60 0x7f6ce0032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f6ce40096a0 tx=0x7f6ce4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:10.677 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.676+0000 7f6ceb7fe700 1 -- 172.21.15.178:0/128052321 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6cdc0095f0 con 0x7f6cf40f53f0 2023-11-15T04:59:10.883 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.882+0000 7f6cf9882700 1 -- 172.21.15.178:0/128052321 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6cf4107010 con 0x7f6cf40f53f0 2023-11-15T04:59:10.883 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.883+0000 7f6ceb7fe700 1 -- 172.21.15.178:0/128052321 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6cdc01db70 con 0x7f6cf40f53f0 2023-11-15T04:59:10.884 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:59:10.884 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:59:10.885 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.885+0000 7f6cf9882700 1 -- 172.21.15.178:0/128052321 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6ce002ff60 msgr2=0x7f6ce0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:10.885 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.885+0000 7f6cf9882700 1 --2- 172.21.15.178:0/128052321 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6ce002ff60 0x7f6ce0032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f6ce40096a0 tx=0x7f6ce4006b40).stop 2023-11-15T04:59:10.885 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.885+0000 7f6cf9882700 1 -- 172.21.15.178:0/128052321 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cf40f53f0 msgr2=0x7f6cf41055e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:10.886 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.885+0000 7f6cf9882700 1 --2- 172.21.15.178:0/128052321 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cf40f53f0 0x7f6cf41055e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f6cdc013510 tx=0x7f6cdc00b040).stop 2023-11-15T04:59:10.886 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.885+0000 7f6cf9882700 1 -- 172.21.15.178:0/128052321 shutdown_connections 2023-11-15T04:59:10.886 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.885+0000 7f6cf9882700 1 --2- 172.21.15.178:0/128052321 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6ce002ff60 0x7f6ce0032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:10.886 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.885+0000 7f6cf9882700 1 --2- 172.21.15.178:0/128052321 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cf40f53f0 0x7f6cf41055e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:10.886 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.885+0000 7f6cf9882700 1 -- 172.21.15.178:0/128052321 >> 172.21.15.178:0/128052321 conn(0x7f6cf40f0b70 msgr2=0x7f6cf40f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:10.886 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.885+0000 7f6cf9882700 1 -- 172.21.15.178:0/128052321 shutdown_connections 2023-11-15T04:59:10.887 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:10.885+0000 7f6cf9882700 1 -- 172.21.15.178:0/128052321 wait complete. 2023-11-15T04:59:10.887 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:11.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:10 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:10.883869+0000 mon.smithi163 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.178:0/128052321' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:59:12.272 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:59:12.273 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:59:12.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:11 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:10.343572+0000 mgr.smithi163.qesyrx (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:13.476 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.475+0000 7ff3b41ce700 1 -- 172.21.15.178:0/2501181938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3ac060220 msgr2=0x7ff3ac0fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:13.477 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.475+0000 7ff3b41ce700 1 --2- 172.21.15.178:0/2501181938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3ac060220 0x7ff3ac0fb550 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7ff3a8009fb0 tx=0x7ff3a8009b30).stop 2023-11-15T04:59:13.477 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.475+0000 7ff3b41ce700 1 -- 172.21.15.178:0/2501181938 shutdown_connections 2023-11-15T04:59:13.477 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.475+0000 7ff3b41ce700 1 --2- 172.21.15.178:0/2501181938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3ac060220 0x7ff3ac0fb550 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:13.477 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.475+0000 7ff3b41ce700 1 -- 172.21.15.178:0/2501181938 >> 172.21.15.178:0/2501181938 conn(0x7ff3ac0f0b70 msgr2=0x7ff3ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:13.478 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.475+0000 7ff3b41ce700 1 -- 172.21.15.178:0/2501181938 shutdown_connections 2023-11-15T04:59:13.478 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.475+0000 7ff3b41ce700 1 -- 172.21.15.178:0/2501181938 wait complete. 2023-11-15T04:59:13.478 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.475+0000 7ff3b41ce700 1 Processor -- start 2023-11-15T04:59:13.478 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.475+0000 7ff3b41ce700 1 -- start start 2023-11-15T04:59:13.480 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.475+0000 7ff3b41ce700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3ac060220 0x7ff3ac1088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:13.480 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.475+0000 7ff3b41ce700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac108da0 con 0x7ff3ac060220 2023-11-15T04:59:13.480 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.476+0000 7ff3b1f6a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3ac060220 0x7ff3ac1088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:13.480 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.476+0000 7ff3b1f6a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3ac060220 0x7ff3ac1088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:57448/0 (socket says 172.21.15.178:57448) 2023-11-15T04:59:13.481 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.476+0000 7ff3b1f6a700 1 -- 172.21.15.178:0/3906764705 learned_addr learned my addr 172.21.15.178:0/3906764705 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:59:13.481 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.476+0000 7ff3b1f6a700 1 -- 172.21.15.178:0/3906764705 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a8005040 con 0x7ff3ac060220 2023-11-15T04:59:13.481 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.476+0000 7ff3b1f6a700 1 --2- 172.21.15.178:0/3906764705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3ac060220 0x7ff3ac1088b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7ff3a8013510 tx=0x7ff3a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:13.481 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.476+0000 7ff3a27fc700 1 -- 172.21.15.178:0/3906764705 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3a8009320 con 0x7ff3ac060220 2023-11-15T04:59:13.481 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.476+0000 7ff3b41ce700 1 -- 172.21.15.178:0/3906764705 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ac108fa0 con 0x7ff3ac060220 2023-11-15T04:59:13.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.476+0000 7ff3a27fc700 1 -- 172.21.15.178:0/3906764705 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3a8013f00 con 0x7ff3ac060220 2023-11-15T04:59:13.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.476+0000 7ff3a27fc700 1 -- 172.21.15.178:0/3906764705 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3a8014dc0 con 0x7ff3ac060220 2023-11-15T04:59:13.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.476+0000 7ff3b41ce700 1 -- 172.21.15.178:0/3906764705 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ac109430 con 0x7ff3ac060220 2023-11-15T04:59:13.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.477+0000 7ff3a27fc700 1 -- 172.21.15.178:0/3906764705 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3a801d3d0 con 0x7ff3ac060220 2023-11-15T04:59:13.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.477+0000 7ff3a27fc700 1 --2- 172.21.15.178:0/3906764705 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff39802ff60 0x7ff398032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:13.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.477+0000 7ff3a27fc700 1 -- 172.21.15.178:0/3906764705 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff3a8041060 con 0x7ff3ac060220 2023-11-15T04:59:13.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.477+0000 7ff3b41ce700 1 -- 172.21.15.178:0/3906764705 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ac045ad0 con 0x7ff3ac060220 2023-11-15T04:59:13.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.477+0000 7ff3b1769700 1 --2- 172.21.15.178:0/3906764705 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff39802ff60 0x7ff398032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:13.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.478+0000 7ff3b1769700 1 --2- 172.21.15.178:0/3906764705 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff39802ff60 0x7ff398032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7ff39c0096a0 tx=0x7ff39c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:13.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.482+0000 7ff3a27fc700 1 -- 172.21.15.178:0/3906764705 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3a8012920 con 0x7ff3ac060220 2023-11-15T04:59:13.685 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.684+0000 7ff3b41ce700 1 -- 172.21.15.178:0/3906764705 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3ac10a180 con 0x7ff3ac060220 2023-11-15T04:59:13.686 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.684+0000 7ff3a27fc700 1 -- 172.21.15.178:0/3906764705 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff3a802b020 con 0x7ff3ac060220 2023-11-15T04:59:13.686 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:59:13.686 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:59:13.687 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.686+0000 7ff3b41ce700 1 -- 172.21.15.178:0/3906764705 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff39802ff60 msgr2=0x7ff398032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:13.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.686+0000 7ff3b41ce700 1 --2- 172.21.15.178:0/3906764705 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff39802ff60 0x7ff398032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7ff39c0096a0 tx=0x7ff39c006b40).stop 2023-11-15T04:59:13.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.686+0000 7ff3b41ce700 1 -- 172.21.15.178:0/3906764705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3ac060220 msgr2=0x7ff3ac1088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:13.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.686+0000 7ff3b41ce700 1 --2- 172.21.15.178:0/3906764705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3ac060220 0x7ff3ac1088b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7ff3a8013510 tx=0x7ff3a800b040).stop 2023-11-15T04:59:13.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.686+0000 7ff3b41ce700 1 -- 172.21.15.178:0/3906764705 shutdown_connections 2023-11-15T04:59:13.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.686+0000 7ff3b41ce700 1 --2- 172.21.15.178:0/3906764705 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff39802ff60 0x7ff398032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:13.689 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.686+0000 7ff3b41ce700 1 --2- 172.21.15.178:0/3906764705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3ac060220 0x7ff3ac1088b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:13.689 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.686+0000 7ff3b41ce700 1 -- 172.21.15.178:0/3906764705 >> 172.21.15.178:0/3906764705 conn(0x7ff3ac0f0b70 msgr2=0x7ff3ac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:13.689 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.686+0000 7ff3b41ce700 1 -- 172.21.15.178:0/3906764705 shutdown_connections 2023-11-15T04:59:13.689 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:13.686+0000 7ff3b41ce700 1 -- 172.21.15.178:0/3906764705 wait complete. 2023-11-15T04:59:13.689 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:14.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:13 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:12.343876+0000 mgr.smithi163.qesyrx (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:14.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:13 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:13.686007+0000 mon.smithi163 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.178:0/3906764705' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:59:15.070 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:59:15.071 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:59:16.221 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.219+0000 7fe59b59e700 1 -- 172.21.15.178:0/1344755078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe59c0f51a0 msgr2=0x7fe59c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:16.222 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.219+0000 7fe59b59e700 1 --2- 172.21.15.178:0/1344755078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe59c0f51a0 0x7fe59c0f55c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fe58c009fb0 tx=0x7fe58c009b30).stop 2023-11-15T04:59:16.222 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.219+0000 7fe59b59e700 1 -- 172.21.15.178:0/1344755078 shutdown_connections 2023-11-15T04:59:16.222 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.219+0000 7fe59b59e700 1 --2- 172.21.15.178:0/1344755078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe59c0f51a0 0x7fe59c0f55c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:16.222 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.219+0000 7fe59b59e700 1 -- 172.21.15.178:0/1344755078 >> 172.21.15.178:0/1344755078 conn(0x7fe59c0f09a0 msgr2=0x7fe59c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:16.222 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.219+0000 7fe59b59e700 1 -- 172.21.15.178:0/1344755078 shutdown_connections 2023-11-15T04:59:16.223 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.219+0000 7fe59b59e700 1 -- 172.21.15.178:0/1344755078 wait complete. 2023-11-15T04:59:16.223 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.219+0000 7fe59b59e700 1 Processor -- start 2023-11-15T04:59:16.223 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.219+0000 7fe59b59e700 1 -- start start 2023-11-15T04:59:16.223 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.219+0000 7fe59b59e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe59c0f51a0 0x7fe59c1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:16.223 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.219+0000 7fe59b59e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c1058a0 con 0x7fe59c0f51a0 2023-11-15T04:59:16.224 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.220+0000 7fe59a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe59c0f51a0 0x7fe59c1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:16.224 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.220+0000 7fe59a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe59c0f51a0 0x7fe59c1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:52842/0 (socket says 172.21.15.178:52842) 2023-11-15T04:59:16.224 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.220+0000 7fe59a59c700 1 -- 172.21.15.178:0/3069921976 learned_addr learned my addr 172.21.15.178:0/3069921976 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:59:16.224 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.220+0000 7fe59a59c700 1 -- 172.21.15.178:0/3069921976 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58c005040 con 0x7fe59c0f51a0 2023-11-15T04:59:16.224 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.220+0000 7fe59a59c700 1 --2- 172.21.15.178:0/3069921976 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe59c0f51a0 0x7fe59c1053b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fe58c013510 tx=0x7fe58c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:16.225 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.220+0000 7fe58affd700 1 -- 172.21.15.178:0/3069921976 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe58c009320 con 0x7fe59c0f51a0 2023-11-15T04:59:16.225 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.220+0000 7fe59b59e700 1 -- 172.21.15.178:0/3069921976 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe59c105aa0 con 0x7fe59c0f51a0 2023-11-15T04:59:16.225 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.220+0000 7fe58affd700 1 -- 172.21.15.178:0/3069921976 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe58c013f00 con 0x7fe59c0f51a0 2023-11-15T04:59:16.225 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.220+0000 7fe58affd700 1 -- 172.21.15.178:0/3069921976 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe58c014dc0 con 0x7fe59c0f51a0 2023-11-15T04:59:16.225 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.220+0000 7fe59b59e700 1 -- 172.21.15.178:0/3069921976 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe59c105f30 con 0x7fe59c0f51a0 2023-11-15T04:59:16.228 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.221+0000 7fe58affd700 1 -- 172.21.15.178:0/3069921976 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe58c014420 con 0x7fe59c0f51a0 2023-11-15T04:59:16.228 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.221+0000 7fe59b59e700 1 -- 172.21.15.178:0/3069921976 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe59c045ad0 con 0x7fe59c0f51a0 2023-11-15T04:59:16.228 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.221+0000 7fe58affd700 1 --2- 172.21.15.178:0/3069921976 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe58402ff60 0x7fe584032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:16.229 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.221+0000 7fe58affd700 1 -- 172.21.15.178:0/3069921976 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe58c041050 con 0x7fe59c0f51a0 2023-11-15T04:59:16.229 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.221+0000 7fe599d9b700 1 --2- 172.21.15.178:0/3069921976 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe58402ff60 0x7fe584032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:16.229 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.222+0000 7fe599d9b700 1 --2- 172.21.15.178:0/3069921976 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe58402ff60 0x7fe584032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fe5900096a0 tx=0x7fe590006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:16.230 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.226+0000 7fe58affd700 1 -- 172.21.15.178:0/3069921976 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe58c0095f0 con 0x7fe59c0f51a0 2023-11-15T04:59:16.434 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.433+0000 7fe59b59e700 1 -- 172.21.15.178:0/3069921976 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe59c106de0 con 0x7fe59c0f51a0 2023-11-15T04:59:16.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.434+0000 7fe58affd700 1 -- 172.21.15.178:0/3069921976 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe58c01cb10 con 0x7fe59c0f51a0 2023-11-15T04:59:16.435 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:59:16.436 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:59:16.437 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.435+0000 7fe59b59e700 1 -- 172.21.15.178:0/3069921976 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe58402ff60 msgr2=0x7fe584032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:16.437 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.435+0000 7fe59b59e700 1 --2- 172.21.15.178:0/3069921976 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe58402ff60 0x7fe584032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fe5900096a0 tx=0x7fe590006b40).stop 2023-11-15T04:59:16.437 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.435+0000 7fe59b59e700 1 -- 172.21.15.178:0/3069921976 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe59c0f51a0 msgr2=0x7fe59c1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:16.437 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.435+0000 7fe59b59e700 1 --2- 172.21.15.178:0/3069921976 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe59c0f51a0 0x7fe59c1053b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fe58c013510 tx=0x7fe58c00b040).stop 2023-11-15T04:59:16.437 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.435+0000 7fe59b59e700 1 -- 172.21.15.178:0/3069921976 shutdown_connections 2023-11-15T04:59:16.438 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.435+0000 7fe59b59e700 1 --2- 172.21.15.178:0/3069921976 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe58402ff60 0x7fe584032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:16.438 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.435+0000 7fe59b59e700 1 --2- 172.21.15.178:0/3069921976 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe59c0f51a0 0x7fe59c1053b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:16.438 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.435+0000 7fe59b59e700 1 -- 172.21.15.178:0/3069921976 >> 172.21.15.178:0/3069921976 conn(0x7fe59c0f09a0 msgr2=0x7fe59c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:16.438 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.435+0000 7fe59b59e700 1 -- 172.21.15.178:0/3069921976 shutdown_connections 2023-11-15T04:59:16.439 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:16.435+0000 7fe59b59e700 1 -- 172.21.15.178:0/3069921976 wait complete. 2023-11-15T04:59:16.439 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:16.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:15 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:14.344213+0000 mgr.smithi163.qesyrx (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:17.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:16.435346+0000 mon.smithi163 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.178:0/3069921976' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:59:17.888 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:59:17.889 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:59:18.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:17 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:16.344595+0000 mgr.smithi163.qesyrx (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:19.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.094+0000 7f271336b700 1 -- 172.21.15.178:0/321563695 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c0f5410 msgr2=0x7f270c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:19.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.094+0000 7f271336b700 1 --2- 172.21.15.178:0/321563695 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c0f5410 0x7f270c0f5830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f2708009fb0 tx=0x7f2708009b30).stop 2023-11-15T04:59:19.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.094+0000 7f271336b700 1 -- 172.21.15.178:0/321563695 shutdown_connections 2023-11-15T04:59:19.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.094+0000 7f271336b700 1 --2- 172.21.15.178:0/321563695 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c0f5410 0x7f270c0f5830 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:19.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.094+0000 7f271336b700 1 -- 172.21.15.178:0/321563695 >> 172.21.15.178:0/321563695 conn(0x7f270c0f0b70 msgr2=0x7f270c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:19.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.094+0000 7f271336b700 1 -- 172.21.15.178:0/321563695 shutdown_connections 2023-11-15T04:59:19.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.094+0000 7f271336b700 1 -- 172.21.15.178:0/321563695 wait complete. 2023-11-15T04:59:19.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.094+0000 7f271336b700 1 Processor -- start 2023-11-15T04:59:19.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.094+0000 7f271336b700 1 -- start start 2023-11-15T04:59:19.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.094+0000 7f271336b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c0f5410 0x7f270c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:19.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.094+0000 7f271336b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270c106c20 con 0x7f270c0f5410 2023-11-15T04:59:19.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.095+0000 7f2711107700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c0f5410 0x7f270c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:19.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.095+0000 7f2711107700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c0f5410 0x7f270c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:52864/0 (socket says 172.21.15.178:52864) 2023-11-15T04:59:19.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.095+0000 7f2711107700 1 -- 172.21.15.178:0/3363501636 learned_addr learned my addr 172.21.15.178:0/3363501636 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:59:19.100 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.095+0000 7f2711107700 1 -- 172.21.15.178:0/3363501636 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2708005040 con 0x7f270c0f5410 2023-11-15T04:59:19.100 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.095+0000 7f2711107700 1 --2- 172.21.15.178:0/3363501636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c0f5410 0x7f270c106730 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f2708013510 tx=0x7f270800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:19.100 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.095+0000 7f2701ffb700 1 -- 172.21.15.178:0/3363501636 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2708009320 con 0x7f270c0f5410 2023-11-15T04:59:19.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.095+0000 7f271336b700 1 -- 172.21.15.178:0/3363501636 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f270c106e20 con 0x7f270c0f5410 2023-11-15T04:59:19.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.095+0000 7f2701ffb700 1 -- 172.21.15.178:0/3363501636 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2708013f00 con 0x7f270c0f5410 2023-11-15T04:59:19.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.095+0000 7f2701ffb700 1 -- 172.21.15.178:0/3363501636 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2708014d90 con 0x7f270c0f5410 2023-11-15T04:59:19.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.096+0000 7f271336b700 1 -- 172.21.15.178:0/3363501636 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f270c1072b0 con 0x7f270c0f5410 2023-11-15T04:59:19.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.096+0000 7f2701ffb700 1 -- 172.21.15.178:0/3363501636 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2708014420 con 0x7f270c0f5410 2023-11-15T04:59:19.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.096+0000 7f271336b700 1 -- 172.21.15.178:0/3363501636 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f270c045ad0 con 0x7f270c0f5410 2023-11-15T04:59:19.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.097+0000 7f2701ffb700 1 --2- 172.21.15.178:0/3363501636 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f26f802ff60 0x7f26f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:19.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.097+0000 7f2701ffb700 1 -- 172.21.15.178:0/3363501636 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f27080410c0 con 0x7f270c0f5410 2023-11-15T04:59:19.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.097+0000 7f2710906700 1 --2- 172.21.15.178:0/3363501636 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f26f802ff60 0x7f26f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:19.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.097+0000 7f2710906700 1 --2- 172.21.15.178:0/3363501636 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f26f802ff60 0x7f26f8032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f26fc0096a0 tx=0x7f26fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:19.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.101+0000 7f2701ffb700 1 -- 172.21.15.178:0/3363501636 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f27080095f0 con 0x7f270c0f5410 2023-11-15T04:59:19.306 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.305+0000 7f271336b700 1 -- 172.21.15.178:0/3363501636 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f270c0fae60 con 0x7f270c0f5410 2023-11-15T04:59:19.307 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.305+0000 7f2701ffb700 1 -- 172.21.15.178:0/3363501636 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f270801db70 con 0x7f270c0f5410 2023-11-15T04:59:19.307 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:59:19.307 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:59:19.308 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.307+0000 7f271336b700 1 -- 172.21.15.178:0/3363501636 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f26f802ff60 msgr2=0x7f26f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:19.308 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.307+0000 7f271336b700 1 --2- 172.21.15.178:0/3363501636 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f26f802ff60 0x7f26f8032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f26fc0096a0 tx=0x7f26fc006b40).stop 2023-11-15T04:59:19.309 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.307+0000 7f271336b700 1 -- 172.21.15.178:0/3363501636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c0f5410 msgr2=0x7f270c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:19.309 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.307+0000 7f271336b700 1 --2- 172.21.15.178:0/3363501636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c0f5410 0x7f270c106730 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f2708013510 tx=0x7f270800b040).stop 2023-11-15T04:59:19.309 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.307+0000 7f271336b700 1 -- 172.21.15.178:0/3363501636 shutdown_connections 2023-11-15T04:59:19.309 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.307+0000 7f271336b700 1 --2- 172.21.15.178:0/3363501636 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f26f802ff60 0x7f26f8032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:19.309 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.307+0000 7f271336b700 1 --2- 172.21.15.178:0/3363501636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c0f5410 0x7f270c106730 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:19.310 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.307+0000 7f271336b700 1 -- 172.21.15.178:0/3363501636 >> 172.21.15.178:0/3363501636 conn(0x7f270c0f0b70 msgr2=0x7f270c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:19.310 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.307+0000 7f271336b700 1 -- 172.21.15.178:0/3363501636 shutdown_connections 2023-11-15T04:59:19.310 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:19.307+0000 7f271336b700 1 -- 172.21.15.178:0/3363501636 wait complete. 2023-11-15T04:59:19.311 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:20.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:19 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:18.344941+0000 mgr.smithi163.qesyrx (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:20.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:19 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:19.307237+0000 mon.smithi163 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.178:0/3363501636' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:59:20.722 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:59:20.723 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:59:21.961 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.959+0000 7f240888c700 1 -- 172.21.15.178:0/400106240 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24040f51a0 msgr2=0x7f24040f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:21.961 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.959+0000 7f240888c700 1 --2- 172.21.15.178:0/400106240 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24040f51a0 0x7f24040f55c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f23f4009fb0 tx=0x7f23f4009b30).stop 2023-11-15T04:59:21.961 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.959+0000 7f240888c700 1 -- 172.21.15.178:0/400106240 shutdown_connections 2023-11-15T04:59:21.961 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.959+0000 7f240888c700 1 --2- 172.21.15.178:0/400106240 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24040f51a0 0x7f24040f55c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:21.962 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.959+0000 7f240888c700 1 -- 172.21.15.178:0/400106240 >> 172.21.15.178:0/400106240 conn(0x7f24040f09a0 msgr2=0x7f24040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:21.962 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.959+0000 7f240888c700 1 -- 172.21.15.178:0/400106240 shutdown_connections 2023-11-15T04:59:21.962 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.959+0000 7f240888c700 1 -- 172.21.15.178:0/400106240 wait complete. 2023-11-15T04:59:21.962 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.959+0000 7f240888c700 1 Processor -- start 2023-11-15T04:59:21.962 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.959+0000 7f240888c700 1 -- start start 2023-11-15T04:59:21.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.959+0000 7f240888c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24040f51a0 0x7f24040fb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:21.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.959+0000 7f240888c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24040f9830 con 0x7f24040f51a0 2023-11-15T04:59:21.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.960+0000 7f24037fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24040f51a0 0x7f24040fb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:21.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.960+0000 7f24037fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24040f51a0 0x7f24040fb3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:52888/0 (socket says 172.21.15.178:52888) 2023-11-15T04:59:21.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.960+0000 7f24037fe700 1 -- 172.21.15.178:0/775811163 learned_addr learned my addr 172.21.15.178:0/775811163 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:59:21.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.960+0000 7f24037fe700 1 -- 172.21.15.178:0/775811163 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f4005040 con 0x7f24040f51a0 2023-11-15T04:59:21.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.960+0000 7f24037fe700 1 --2- 172.21.15.178:0/775811163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24040f51a0 0x7f24040fb3b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f23f4013510 tx=0x7f23f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:21.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.960+0000 7f23ebfff700 1 -- 172.21.15.178:0/775811163 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23f4009320 con 0x7f24040f51a0 2023-11-15T04:59:21.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.960+0000 7f240888c700 1 -- 172.21.15.178:0/775811163 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24040f99d0 con 0x7f24040f51a0 2023-11-15T04:59:21.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.960+0000 7f23ebfff700 1 -- 172.21.15.178:0/775811163 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23f4013f00 con 0x7f24040f51a0 2023-11-15T04:59:21.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.960+0000 7f23ebfff700 1 -- 172.21.15.178:0/775811163 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23f4014dc0 con 0x7f24040f51a0 2023-11-15T04:59:21.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.960+0000 7f240888c700 1 -- 172.21.15.178:0/775811163 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24040f9e60 con 0x7f24040f51a0 2023-11-15T04:59:21.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.961+0000 7f23ebfff700 1 -- 172.21.15.178:0/775811163 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f23f4014420 con 0x7f24040f51a0 2023-11-15T04:59:21.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.961+0000 7f240888c700 1 -- 172.21.15.178:0/775811163 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e4004fa0 con 0x7f24040f51a0 2023-11-15T04:59:21.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.961+0000 7f23ebfff700 1 --2- 172.21.15.178:0/775811163 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f23ec02ff60 0x7f23ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:21.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.961+0000 7f23ebfff700 1 -- 172.21.15.178:0/775811163 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f23f4040ff0 con 0x7f24040f51a0 2023-11-15T04:59:21.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.962+0000 7f2402ffd700 1 --2- 172.21.15.178:0/775811163 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f23ec02ff60 0x7f23ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:21.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.962+0000 7f2402ffd700 1 --2- 172.21.15.178:0/775811163 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f23ec02ff60 0x7f23ec032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f23f80096a0 tx=0x7f23f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:21.968 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:21.966+0000 7f23ebfff700 1 -- 172.21.15.178:0/775811163 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23f40095f0 con 0x7f24040f51a0 2023-11-15T04:59:22.174 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:22.172+0000 7f240888c700 1 -- 172.21.15.178:0/775811163 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23e4005670 con 0x7f24040f51a0 2023-11-15T04:59:22.174 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:22.173+0000 7f23ebfff700 1 -- 172.21.15.178:0/775811163 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f23f401cb10 con 0x7f24040f51a0 2023-11-15T04:59:22.175 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:59:22.175 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:59:22.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:22.175+0000 7f240888c700 1 -- 172.21.15.178:0/775811163 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f23ec02ff60 msgr2=0x7f23ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:22.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:22.175+0000 7f240888c700 1 --2- 172.21.15.178:0/775811163 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f23ec02ff60 0x7f23ec032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f23f80096a0 tx=0x7f23f8006b40).stop 2023-11-15T04:59:22.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:22.175+0000 7f240888c700 1 -- 172.21.15.178:0/775811163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24040f51a0 msgr2=0x7f24040fb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:22.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:22.175+0000 7f240888c700 1 --2- 172.21.15.178:0/775811163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24040f51a0 0x7f24040fb3b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f23f4013510 tx=0x7f23f400b040).stop 2023-11-15T04:59:22.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:22.175+0000 7f240888c700 1 -- 172.21.15.178:0/775811163 shutdown_connections 2023-11-15T04:59:22.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:22.175+0000 7f240888c700 1 --2- 172.21.15.178:0/775811163 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f23ec02ff60 0x7f23ec032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:22.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:22.175+0000 7f240888c700 1 --2- 172.21.15.178:0/775811163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24040f51a0 0x7f24040fb3b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:22.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:22.175+0000 7f240888c700 1 -- 172.21.15.178:0/775811163 >> 172.21.15.178:0/775811163 conn(0x7f24040f09a0 msgr2=0x7f24040f1400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:22.178 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:22.175+0000 7f240888c700 1 -- 172.21.15.178:0/775811163 shutdown_connections 2023-11-15T04:59:22.178 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:22.175+0000 7f240888c700 1 -- 172.21.15.178:0/775811163 wait complete. 2023-11-15T04:59:22.178 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:22.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:21 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:20.345219+0000 mgr.smithi163.qesyrx (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:23.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:22 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:22.174658+0000 mon.smithi163 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.178:0/775811163' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:59:23.574 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:59:23.575 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:59:24.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:23 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:22.345575+0000 mgr.smithi163.qesyrx (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:24.812 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.810+0000 7f532faa9700 1 -- 172.21.15.178:0/944037815 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53280f6510 msgr2=0x7f53280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:24.813 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.810+0000 7f532faa9700 1 --2- 172.21.15.178:0/944037815 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53280f6510 0x7f53280f6930 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f5318009fb0 tx=0x7f5318009b30).stop 2023-11-15T04:59:24.813 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.810+0000 7f532faa9700 1 -- 172.21.15.178:0/944037815 shutdown_connections 2023-11-15T04:59:24.813 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.810+0000 7f532faa9700 1 --2- 172.21.15.178:0/944037815 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53280f6510 0x7f53280f6930 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:24.813 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.810+0000 7f532faa9700 1 -- 172.21.15.178:0/944037815 >> 172.21.15.178:0/944037815 conn(0x7f53280f1c70 msgr2=0x7f53280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:24.814 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.810+0000 7f532faa9700 1 -- 172.21.15.178:0/944037815 shutdown_connections 2023-11-15T04:59:24.814 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.810+0000 7f532faa9700 1 -- 172.21.15.178:0/944037815 wait complete. 2023-11-15T04:59:24.814 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.811+0000 7f532faa9700 1 Processor -- start 2023-11-15T04:59:24.814 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.811+0000 7f532faa9700 1 -- start start 2023-11-15T04:59:24.814 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.811+0000 7f532faa9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53280f6510 0x7f5328106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:24.815 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.811+0000 7f532faa9700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5328106c20 con 0x7f53280f6510 2023-11-15T04:59:24.815 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.811+0000 7f532d845700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53280f6510 0x7f5328106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:24.815 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.811+0000 7f532d845700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53280f6510 0x7f5328106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:52006/0 (socket says 172.21.15.178:52006) 2023-11-15T04:59:24.815 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.811+0000 7f532d845700 1 -- 172.21.15.178:0/2315272817 learned_addr learned my addr 172.21.15.178:0/2315272817 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:59:24.815 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.811+0000 7f532d845700 1 -- 172.21.15.178:0/2315272817 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5318005040 con 0x7f53280f6510 2023-11-15T04:59:24.817 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.812+0000 7f532d845700 1 --2- 172.21.15.178:0/2315272817 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53280f6510 0x7f5328106730 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f5318013510 tx=0x7f531800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:24.817 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.812+0000 7f531e7fc700 1 -- 172.21.15.178:0/2315272817 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5318009320 con 0x7f53280f6510 2023-11-15T04:59:24.817 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.812+0000 7f532faa9700 1 -- 172.21.15.178:0/2315272817 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5328106e20 con 0x7f53280f6510 2023-11-15T04:59:24.818 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.812+0000 7f531e7fc700 1 -- 172.21.15.178:0/2315272817 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5318013f00 con 0x7f53280f6510 2023-11-15T04:59:24.818 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.812+0000 7f531e7fc700 1 -- 172.21.15.178:0/2315272817 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5318014d90 con 0x7f53280f6510 2023-11-15T04:59:24.818 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.812+0000 7f532faa9700 1 -- 172.21.15.178:0/2315272817 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53281072b0 con 0x7f53280f6510 2023-11-15T04:59:24.818 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.813+0000 7f531e7fc700 1 -- 172.21.15.178:0/2315272817 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5318014420 con 0x7f53280f6510 2023-11-15T04:59:24.818 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.813+0000 7f532faa9700 1 -- 172.21.15.178:0/2315272817 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5328045ad0 con 0x7f53280f6510 2023-11-15T04:59:24.819 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.813+0000 7f531e7fc700 1 --2- 172.21.15.178:0/2315272817 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f531402ff60 0x7f5314032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:24.819 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.813+0000 7f531e7fc700 1 -- 172.21.15.178:0/2315272817 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5318040ff0 con 0x7f53280f6510 2023-11-15T04:59:24.819 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.813+0000 7f532d044700 1 --2- 172.21.15.178:0/2315272817 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f531402ff60 0x7f5314032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:24.819 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.813+0000 7f532d044700 1 --2- 172.21.15.178:0/2315272817 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f531402ff60 0x7f5314032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f53240096a0 tx=0x7f5324006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:24.820 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:24.818+0000 7f531e7fc700 1 -- 172.21.15.178:0/2315272817 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f53180095f0 con 0x7f53280f6510 2023-11-15T04:59:25.021 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:25.020+0000 7f532faa9700 1 -- 172.21.15.178:0/2315272817 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53280fbf60 con 0x7f53280f6510 2023-11-15T04:59:25.022 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:25.021+0000 7f531e7fc700 1 -- 172.21.15.178:0/2315272817 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f531801db70 con 0x7f53280f6510 2023-11-15T04:59:25.023 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:59:25.023 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:59:25.024 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:25.022+0000 7f532faa9700 1 -- 172.21.15.178:0/2315272817 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f531402ff60 msgr2=0x7f5314032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:25.024 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:25.022+0000 7f532faa9700 1 --2- 172.21.15.178:0/2315272817 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f531402ff60 0x7f5314032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f53240096a0 tx=0x7f5324006b40).stop 2023-11-15T04:59:25.024 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:25.022+0000 7f532faa9700 1 -- 172.21.15.178:0/2315272817 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53280f6510 msgr2=0x7f5328106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:25.024 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:25.022+0000 7f532faa9700 1 --2- 172.21.15.178:0/2315272817 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53280f6510 0x7f5328106730 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f5318013510 tx=0x7f531800b040).stop 2023-11-15T04:59:25.024 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:25.022+0000 7f532faa9700 1 -- 172.21.15.178:0/2315272817 shutdown_connections 2023-11-15T04:59:25.025 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:25.022+0000 7f532faa9700 1 --2- 172.21.15.178:0/2315272817 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f531402ff60 0x7f5314032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:25.025 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:25.022+0000 7f532faa9700 1 --2- 172.21.15.178:0/2315272817 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53280f6510 0x7f5328106730 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:25.025 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:25.022+0000 7f532faa9700 1 -- 172.21.15.178:0/2315272817 >> 172.21.15.178:0/2315272817 conn(0x7f53280f1c70 msgr2=0x7f53280f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:25.025 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:25.022+0000 7f532faa9700 1 -- 172.21.15.178:0/2315272817 shutdown_connections 2023-11-15T04:59:25.025 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:25.022+0000 7f532faa9700 1 -- 172.21.15.178:0/2315272817 wait complete. 2023-11-15T04:59:25.026 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:25.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:24 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:24.451412+0000 mon.smithi163 (mon.0) 197 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/mirror_snapshot_schedule"}]: dispatch 2023-11-15T04:59:25.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:24 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:24.453063+0000 mon.smithi163 (mon.0) 198 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/trash_purge_schedule"}]: dispatch 2023-11-15T04:59:26.424 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:59:26.425 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:59:26.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:24.345926+0000 mgr.smithi163.qesyrx (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:26.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:25.022375+0000 mon.smithi163 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.178:0/2315272817' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:59:27.635 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.632+0000 7f9ed2244700 1 -- 172.21.15.178:0/474948530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ecc0f6510 msgr2=0x7f9ecc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:27.635 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.632+0000 7f9ed2244700 1 --2- 172.21.15.178:0/474948530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ecc0f6510 0x7f9ecc0f6930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f9ec0009fb0 tx=0x7f9ec0009b30).stop 2023-11-15T04:59:27.635 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.632+0000 7f9ed2244700 1 -- 172.21.15.178:0/474948530 shutdown_connections 2023-11-15T04:59:27.635 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.632+0000 7f9ed2244700 1 --2- 172.21.15.178:0/474948530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ecc0f6510 0x7f9ecc0f6930 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:27.635 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.632+0000 7f9ed2244700 1 -- 172.21.15.178:0/474948530 >> 172.21.15.178:0/474948530 conn(0x7f9ecc0f1c70 msgr2=0x7f9ecc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:27.636 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.632+0000 7f9ed2244700 1 -- 172.21.15.178:0/474948530 shutdown_connections 2023-11-15T04:59:27.636 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.632+0000 7f9ed2244700 1 -- 172.21.15.178:0/474948530 wait complete. 2023-11-15T04:59:27.636 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.633+0000 7f9ed2244700 1 Processor -- start 2023-11-15T04:59:27.636 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.633+0000 7f9ed2244700 1 -- start start 2023-11-15T04:59:27.636 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.633+0000 7f9ed2244700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ecc0f6510 0x7f9ecc06b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:27.637 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.633+0000 7f9ed2244700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc069e20 con 0x7f9ecc0f6510 2023-11-15T04:59:27.637 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.633+0000 7f9ecb7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ecc0f6510 0x7f9ecc06b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:27.637 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.634+0000 7f9ecb7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ecc0f6510 0x7f9ecc06b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:52026/0 (socket says 172.21.15.178:52026) 2023-11-15T04:59:27.637 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.634+0000 7f9ecb7fe700 1 -- 172.21.15.178:0/984047660 learned_addr learned my addr 172.21.15.178:0/984047660 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:59:27.639 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.634+0000 7f9ecb7fe700 1 -- 172.21.15.178:0/984047660 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec0005040 con 0x7f9ecc0f6510 2023-11-15T04:59:27.639 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.634+0000 7f9ecb7fe700 1 --2- 172.21.15.178:0/984047660 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ecc0f6510 0x7f9ecc06b930 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f9ec0013510 tx=0x7f9ec000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:27.639 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.634+0000 7f9eb3fff700 1 -- 172.21.15.178:0/984047660 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ec0009320 con 0x7f9ecc0f6510 2023-11-15T04:59:27.639 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.634+0000 7f9ed2244700 1 -- 172.21.15.178:0/984047660 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ecc06a020 con 0x7f9ecc0f6510 2023-11-15T04:59:27.639 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.634+0000 7f9eb3fff700 1 -- 172.21.15.178:0/984047660 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ec0013f00 con 0x7f9ecc0f6510 2023-11-15T04:59:27.640 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.634+0000 7f9eb3fff700 1 -- 172.21.15.178:0/984047660 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ec0014d90 con 0x7f9ecc0f6510 2023-11-15T04:59:27.640 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.634+0000 7f9ed2244700 1 -- 172.21.15.178:0/984047660 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ecc06a4b0 con 0x7f9ecc0f6510 2023-11-15T04:59:27.640 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.635+0000 7f9eb3fff700 1 -- 172.21.15.178:0/984047660 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ec00263a0 con 0x7f9ecc0f6510 2023-11-15T04:59:27.640 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.635+0000 7f9eb3fff700 1 --2- 172.21.15.178:0/984047660 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f9eb402ff60 0x7f9eb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:27.640 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.635+0000 7f9eb3fff700 1 -- 172.21.15.178:0/984047660 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ec0041f70 con 0x7f9ecc0f6510 2023-11-15T04:59:27.641 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.635+0000 7f9ed2244700 1 -- 172.21.15.178:0/984047660 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ecc059100 con 0x7f9ecc0f6510 2023-11-15T04:59:27.641 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.635+0000 7f9ecaffd700 1 --2- 172.21.15.178:0/984047660 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f9eb402ff60 0x7f9eb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:27.641 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.636+0000 7f9ecaffd700 1 --2- 172.21.15.178:0/984047660 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f9eb402ff60 0x7f9eb4032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f9ebc0096a0 tx=0x7f9ebc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:27.641 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.640+0000 7f9eb3fff700 1 -- 172.21.15.178:0/984047660 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ec0012920 con 0x7f9ecc0f6510 2023-11-15T04:59:27.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.849+0000 7f9ed2244700 1 -- 172.21.15.178:0/984047660 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ecc045ad0 con 0x7f9ecc0f6510 2023-11-15T04:59:27.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.850+0000 7f9eb3fff700 1 -- 172.21.15.178:0/984047660 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9ec002b020 con 0x7f9ecc0f6510 2023-11-15T04:59:27.852 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:59:27.852 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:59:27.853 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.852+0000 7f9ed2244700 1 -- 172.21.15.178:0/984047660 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f9eb402ff60 msgr2=0x7f9eb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:27.853 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.852+0000 7f9ed2244700 1 --2- 172.21.15.178:0/984047660 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f9eb402ff60 0x7f9eb4032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f9ebc0096a0 tx=0x7f9ebc006b40).stop 2023-11-15T04:59:27.853 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.852+0000 7f9ed2244700 1 -- 172.21.15.178:0/984047660 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ecc0f6510 msgr2=0x7f9ecc06b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:27.854 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.852+0000 7f9ed2244700 1 --2- 172.21.15.178:0/984047660 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ecc0f6510 0x7f9ecc06b930 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f9ec0013510 tx=0x7f9ec000b040).stop 2023-11-15T04:59:27.854 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.852+0000 7f9ed2244700 1 -- 172.21.15.178:0/984047660 shutdown_connections 2023-11-15T04:59:27.854 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.852+0000 7f9ed2244700 1 --2- 172.21.15.178:0/984047660 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f9eb402ff60 0x7f9eb4032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:27.854 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.852+0000 7f9ed2244700 1 --2- 172.21.15.178:0/984047660 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ecc0f6510 0x7f9ecc06b930 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:27.854 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.852+0000 7f9ed2244700 1 -- 172.21.15.178:0/984047660 >> 172.21.15.178:0/984047660 conn(0x7f9ecc0f1c70 msgr2=0x7f9ecc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:27.855 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.852+0000 7f9ed2244700 1 -- 172.21.15.178:0/984047660 shutdown_connections 2023-11-15T04:59:27.855 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:27.852+0000 7f9ed2244700 1 -- 172.21.15.178:0/984047660 wait complete. 2023-11-15T04:59:27.855 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:28.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:27 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:26.346310+0000 mgr.smithi163.qesyrx (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:28.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:27 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:27.851707+0000 mon.smithi163 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.178:0/984047660' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:59:29.254 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:59:29.254 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:59:30.428 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.425+0000 7f7fe159e700 1 -- 172.21.15.178:0/295174533 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7fdc0f51a0 msgr2=0x7f7fdc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:30.429 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.425+0000 7f7fe159e700 1 --2- 172.21.15.178:0/295174533 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7fdc0f51a0 0x7f7fdc0f55c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f7fcc009fb0 tx=0x7f7fcc009b30).stop 2023-11-15T04:59:30.429 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.425+0000 7f7fe159e700 1 -- 172.21.15.178:0/295174533 shutdown_connections 2023-11-15T04:59:30.429 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.425+0000 7f7fe159e700 1 --2- 172.21.15.178:0/295174533 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7fdc0f51a0 0x7f7fdc0f55c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:30.430 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.425+0000 7f7fe159e700 1 -- 172.21.15.178:0/295174533 >> 172.21.15.178:0/295174533 conn(0x7f7fdc0f09a0 msgr2=0x7f7fdc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:30.430 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.425+0000 7f7fe159e700 1 -- 172.21.15.178:0/295174533 shutdown_connections 2023-11-15T04:59:30.430 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.425+0000 7f7fe159e700 1 -- 172.21.15.178:0/295174533 wait complete. 2023-11-15T04:59:30.430 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.426+0000 7f7fe159e700 1 Processor -- start 2023-11-15T04:59:30.430 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.426+0000 7f7fe159e700 1 -- start start 2023-11-15T04:59:30.431 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.426+0000 7f7fe159e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7fdc0f51a0 0x7f7fdc1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:30.431 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.426+0000 7f7fe159e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc1058a0 con 0x7f7fdc0f51a0 2023-11-15T04:59:30.431 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.427+0000 7f7fdbfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7fdc0f51a0 0x7f7fdc1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:30.431 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.427+0000 7f7fdbfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7fdc0f51a0 0x7f7fdc1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:52034/0 (socket says 172.21.15.178:52034) 2023-11-15T04:59:30.432 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.427+0000 7f7fdbfff700 1 -- 172.21.15.178:0/3291574806 learned_addr learned my addr 172.21.15.178:0/3291574806 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:59:30.432 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.427+0000 7f7fdbfff700 1 -- 172.21.15.178:0/3291574806 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fcc005040 con 0x7f7fdc0f51a0 2023-11-15T04:59:30.432 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.427+0000 7f7fdbfff700 1 --2- 172.21.15.178:0/3291574806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7fdc0f51a0 0x7f7fdc1053b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f7fcc013510 tx=0x7f7fcc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:30.432 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.427+0000 7f7fd8ff9700 1 -- 172.21.15.178:0/3291574806 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fcc009320 con 0x7f7fdc0f51a0 2023-11-15T04:59:30.432 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.427+0000 7f7fe159e700 1 -- 172.21.15.178:0/3291574806 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fdc105aa0 con 0x7f7fdc0f51a0 2023-11-15T04:59:30.433 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.427+0000 7f7fd8ff9700 1 -- 172.21.15.178:0/3291574806 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7fcc013f00 con 0x7f7fdc0f51a0 2023-11-15T04:59:30.433 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.427+0000 7f7fd8ff9700 1 -- 172.21.15.178:0/3291574806 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fcc014dc0 con 0x7f7fdc0f51a0 2023-11-15T04:59:30.433 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.427+0000 7f7fe159e700 1 -- 172.21.15.178:0/3291574806 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fdc105f30 con 0x7f7fdc0f51a0 2023-11-15T04:59:30.434 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.428+0000 7f7fd8ff9700 1 -- 172.21.15.178:0/3291574806 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7fcc014420 con 0x7f7fdc0f51a0 2023-11-15T04:59:30.434 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.428+0000 7f7fe159e700 1 -- 172.21.15.178:0/3291574806 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fdc045ad0 con 0x7f7fdc0f51a0 2023-11-15T04:59:30.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.428+0000 7f7fd8ff9700 1 --2- 172.21.15.178:0/3291574806 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7fc402ff60 0x7f7fc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:30.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.428+0000 7f7fd8ff9700 1 -- 172.21.15.178:0/3291574806 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7fcc041070 con 0x7f7fdc0f51a0 2023-11-15T04:59:30.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.428+0000 7f7fdb7fe700 1 --2- 172.21.15.178:0/3291574806 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7fc402ff60 0x7f7fc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:30.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.429+0000 7f7fdb7fe700 1 --2- 172.21.15.178:0/3291574806 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7fc402ff60 0x7f7fc4032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f7fd00096a0 tx=0x7f7fd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:30.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.433+0000 7f7fd8ff9700 1 -- 172.21.15.178:0/3291574806 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7fcc0095f0 con 0x7f7fdc0f51a0 2023-11-15T04:59:30.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:29 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:28.346701+0000 mgr.smithi163.qesyrx (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:30.642 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.640+0000 7f7fe159e700 1 -- 172.21.15.178:0/3291574806 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7fdc106d80 con 0x7f7fdc0f51a0 2023-11-15T04:59:30.643 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.641+0000 7f7fd8ff9700 1 -- 172.21.15.178:0/3291574806 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7fcc01cb10 con 0x7f7fdc0f51a0 2023-11-15T04:59:30.644 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:59:30.644 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:59:30.645 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.642+0000 7f7fe159e700 1 -- 172.21.15.178:0/3291574806 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7fc402ff60 msgr2=0x7f7fc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:30.645 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.642+0000 7f7fe159e700 1 --2- 172.21.15.178:0/3291574806 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7fc402ff60 0x7f7fc4032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f7fd00096a0 tx=0x7f7fd0006b40).stop 2023-11-15T04:59:30.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.642+0000 7f7fe159e700 1 -- 172.21.15.178:0/3291574806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7fdc0f51a0 msgr2=0x7f7fdc1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:30.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.642+0000 7f7fe159e700 1 --2- 172.21.15.178:0/3291574806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7fdc0f51a0 0x7f7fdc1053b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f7fcc013510 tx=0x7f7fcc00b040).stop 2023-11-15T04:59:30.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.642+0000 7f7fe159e700 1 -- 172.21.15.178:0/3291574806 shutdown_connections 2023-11-15T04:59:30.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.642+0000 7f7fe159e700 1 --2- 172.21.15.178:0/3291574806 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7fc402ff60 0x7f7fc4032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:30.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.642+0000 7f7fe159e700 1 --2- 172.21.15.178:0/3291574806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7fdc0f51a0 0x7f7fdc1053b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:30.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.642+0000 7f7fe159e700 1 -- 172.21.15.178:0/3291574806 >> 172.21.15.178:0/3291574806 conn(0x7f7fdc0f09a0 msgr2=0x7f7fdc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:30.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.642+0000 7f7fe159e700 1 -- 172.21.15.178:0/3291574806 shutdown_connections 2023-11-15T04:59:30.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:30.642+0000 7f7fe159e700 1 -- 172.21.15.178:0/3291574806 wait complete. 2023-11-15T04:59:30.648 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:31.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:30 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:30.642269+0000 mon.smithi163 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.178:0/3291574806' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:59:32.071 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:59:32.072 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:59:32.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:31 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:30.347028+0000 mgr.smithi163.qesyrx (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:33.257 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.254+0000 7ff3f3fff700 1 -- 172.21.15.178:0/2143757410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3f40601e0 msgr2=0x7ff3f40f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:33.257 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.254+0000 7ff3f3fff700 1 --2- 172.21.15.178:0/2143757410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3f40601e0 0x7ff3f40f8f20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7ff3e4009fb0 tx=0x7ff3e4009b30).stop 2023-11-15T04:59:33.257 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.254+0000 7ff3f3fff700 1 -- 172.21.15.178:0/2143757410 shutdown_connections 2023-11-15T04:59:33.257 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.254+0000 7ff3f3fff700 1 --2- 172.21.15.178:0/2143757410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3f40601e0 0x7ff3f40f8f20 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:33.258 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.254+0000 7ff3f3fff700 1 -- 172.21.15.178:0/2143757410 >> 172.21.15.178:0/2143757410 conn(0x7ff3f40f09a0 msgr2=0x7ff3f40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:33.258 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.255+0000 7ff3f3fff700 1 -- 172.21.15.178:0/2143757410 shutdown_connections 2023-11-15T04:59:33.258 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.255+0000 7ff3f3fff700 1 -- 172.21.15.178:0/2143757410 wait complete. 2023-11-15T04:59:33.258 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.255+0000 7ff3f3fff700 1 Processor -- start 2023-11-15T04:59:33.258 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.255+0000 7ff3f3fff700 1 -- start start 2023-11-15T04:59:33.259 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.255+0000 7ff3f3fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3f40601e0 0x7ff3f4106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:33.259 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.255+0000 7ff3f3fff700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f41069f0 con 0x7ff3f40601e0 2023-11-15T04:59:33.259 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.256+0000 7ff3f2ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3f40601e0 0x7ff3f4106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:33.259 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.256+0000 7ff3f2ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3f40601e0 0x7ff3f4106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:52050/0 (socket says 172.21.15.178:52050) 2023-11-15T04:59:33.259 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.256+0000 7ff3f2ffd700 1 -- 172.21.15.178:0/2104167405 learned_addr learned my addr 172.21.15.178:0/2104167405 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:59:33.261 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.256+0000 7ff3f2ffd700 1 -- 172.21.15.178:0/2104167405 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e4005040 con 0x7ff3f40601e0 2023-11-15T04:59:33.262 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.256+0000 7ff3f2ffd700 1 --2- 172.21.15.178:0/2104167405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3f40601e0 0x7ff3f4106500 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7ff3e4013510 tx=0x7ff3e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:33.262 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.256+0000 7ff3db7fe700 1 -- 172.21.15.178:0/2104167405 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3e4009320 con 0x7ff3f40601e0 2023-11-15T04:59:33.262 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.256+0000 7ff3db7fe700 1 -- 172.21.15.178:0/2104167405 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3e4013f00 con 0x7ff3f40601e0 2023-11-15T04:59:33.262 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.256+0000 7ff3f3fff700 1 -- 172.21.15.178:0/2104167405 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f4106bf0 con 0x7ff3f40601e0 2023-11-15T04:59:33.262 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.256+0000 7ff3db7fe700 1 -- 172.21.15.178:0/2104167405 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3e4014d90 con 0x7ff3f40601e0 2023-11-15T04:59:33.263 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.256+0000 7ff3f3fff700 1 -- 172.21.15.178:0/2104167405 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f4107080 con 0x7ff3f40601e0 2023-11-15T04:59:33.263 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.257+0000 7ff3db7fe700 1 -- 172.21.15.178:0/2104167405 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3e40263a0 con 0x7ff3f40601e0 2023-11-15T04:59:33.263 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.257+0000 7ff3db7fe700 1 --2- 172.21.15.178:0/2104167405 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff3dc02ff60 0x7ff3dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:33.263 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.257+0000 7ff3db7fe700 1 -- 172.21.15.178:0/2104167405 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff3e4041010 con 0x7ff3f40601e0 2023-11-15T04:59:33.264 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.257+0000 7ff3f3fff700 1 -- 172.21.15.178:0/2104167405 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f4045ad0 con 0x7ff3f40601e0 2023-11-15T04:59:33.264 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.257+0000 7ff3f27fc700 1 --2- 172.21.15.178:0/2104167405 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff3dc02ff60 0x7ff3dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:33.264 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.258+0000 7ff3f27fc700 1 --2- 172.21.15.178:0/2104167405 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff3dc02ff60 0x7ff3dc032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7ff3e80096a0 tx=0x7ff3e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:33.265 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.262+0000 7ff3db7fe700 1 -- 172.21.15.178:0/2104167405 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3e4012920 con 0x7ff3f40601e0 2023-11-15T04:59:33.469 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.468+0000 7ff3f3fff700 1 -- 172.21.15.178:0/2104167405 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3f4107da0 con 0x7ff3f40601e0 2023-11-15T04:59:33.470 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.469+0000 7ff3db7fe700 1 -- 172.21.15.178:0/2104167405 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff3e402b020 con 0x7ff3f40601e0 2023-11-15T04:59:33.470 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:59:33.470 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:59:33.472 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.470+0000 7ff3f3fff700 1 -- 172.21.15.178:0/2104167405 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff3dc02ff60 msgr2=0x7ff3dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:33.472 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.470+0000 7ff3f3fff700 1 --2- 172.21.15.178:0/2104167405 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff3dc02ff60 0x7ff3dc032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7ff3e80096a0 tx=0x7ff3e8006b40).stop 2023-11-15T04:59:33.472 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.470+0000 7ff3f3fff700 1 -- 172.21.15.178:0/2104167405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3f40601e0 msgr2=0x7ff3f4106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:33.473 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.470+0000 7ff3f3fff700 1 --2- 172.21.15.178:0/2104167405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3f40601e0 0x7ff3f4106500 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7ff3e4013510 tx=0x7ff3e400b040).stop 2023-11-15T04:59:33.473 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.470+0000 7ff3f3fff700 1 -- 172.21.15.178:0/2104167405 shutdown_connections 2023-11-15T04:59:33.473 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.470+0000 7ff3f3fff700 1 --2- 172.21.15.178:0/2104167405 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff3dc02ff60 0x7ff3dc032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:33.473 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.470+0000 7ff3f3fff700 1 --2- 172.21.15.178:0/2104167405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3f40601e0 0x7ff3f4106500 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:33.473 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.470+0000 7ff3f3fff700 1 -- 172.21.15.178:0/2104167405 >> 172.21.15.178:0/2104167405 conn(0x7ff3f40f09a0 msgr2=0x7ff3f40f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:33.474 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.471+0000 7ff3f3fff700 1 -- 172.21.15.178:0/2104167405 shutdown_connections 2023-11-15T04:59:33.474 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:33.471+0000 7ff3f3fff700 1 -- 172.21.15.178:0/2104167405 wait complete. 2023-11-15T04:59:33.474 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:34.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:33 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:32.347314+0000 mgr.smithi163.qesyrx (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:34.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:33 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:33.470344+0000 mon.smithi163 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.178:0/2104167405' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:59:34.878 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:59:34.879 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:59:36.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.100+0000 7f05bb16c700 1 -- 172.21.15.178:0/440280212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05b40f5410 msgr2=0x7f05b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:36.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.100+0000 7f05bb16c700 1 --2- 172.21.15.178:0/440280212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05b40f5410 0x7f05b40f5830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f05a8009fb0 tx=0x7f05a8009b30).stop 2023-11-15T04:59:36.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.100+0000 7f05bb16c700 1 -- 172.21.15.178:0/440280212 shutdown_connections 2023-11-15T04:59:36.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.100+0000 7f05bb16c700 1 --2- 172.21.15.178:0/440280212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05b40f5410 0x7f05b40f5830 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:36.104 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.100+0000 7f05bb16c700 1 -- 172.21.15.178:0/440280212 >> 172.21.15.178:0/440280212 conn(0x7f05b40f0b70 msgr2=0x7f05b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:36.104 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.100+0000 7f05bb16c700 1 -- 172.21.15.178:0/440280212 shutdown_connections 2023-11-15T04:59:36.104 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.100+0000 7f05bb16c700 1 -- 172.21.15.178:0/440280212 wait complete. 2023-11-15T04:59:36.104 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.101+0000 7f05bb16c700 1 Processor -- start 2023-11-15T04:59:36.104 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.101+0000 7f05bb16c700 1 -- start start 2023-11-15T04:59:36.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.101+0000 7f05bb16c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05b40f5410 0x7f05b4109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:36.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.101+0000 7f05bb16c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b4060220 con 0x7f05b40f5410 2023-11-15T04:59:36.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.101+0000 7f05b8f08700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05b40f5410 0x7f05b4109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:36.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.101+0000 7f05b8f08700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05b40f5410 0x7f05b4109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:37332/0 (socket says 172.21.15.178:37332) 2023-11-15T04:59:36.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.101+0000 7f05b8f08700 1 -- 172.21.15.178:0/3530269065 learned_addr learned my addr 172.21.15.178:0/3530269065 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:59:36.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.102+0000 7f05b8f08700 1 -- 172.21.15.178:0/3530269065 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a8005040 con 0x7f05b40f5410 2023-11-15T04:59:36.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.102+0000 7f05b8f08700 1 --2- 172.21.15.178:0/3530269065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05b40f5410 0x7f05b4109450 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f05a8013510 tx=0x7f05a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:36.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.102+0000 7f05b17fa700 1 -- 172.21.15.178:0/3530269065 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05a8009320 con 0x7f05b40f5410 2023-11-15T04:59:36.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.102+0000 7f05b17fa700 1 -- 172.21.15.178:0/3530269065 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f05a8013f00 con 0x7f05b40f5410 2023-11-15T04:59:36.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.102+0000 7f05bb16c700 1 -- 172.21.15.178:0/3530269065 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b4060420 con 0x7f05b40f5410 2023-11-15T04:59:36.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.102+0000 7f05b17fa700 1 -- 172.21.15.178:0/3530269065 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05a8014d90 con 0x7f05b40f5410 2023-11-15T04:59:36.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.102+0000 7f05bb16c700 1 -- 172.21.15.178:0/3530269065 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b4109b00 con 0x7f05b40f5410 2023-11-15T04:59:36.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.103+0000 7f05b17fa700 1 -- 172.21.15.178:0/3530269065 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f05a80095f0 con 0x7f05b40f5410 2023-11-15T04:59:36.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.103+0000 7f05bb16c700 1 -- 172.21.15.178:0/3530269065 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05b4059100 con 0x7f05b40f5410 2023-11-15T04:59:36.109 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.103+0000 7f05b17fa700 1 --2- 172.21.15.178:0/3530269065 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f05a402ff60 0x7f05a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:36.109 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.103+0000 7f05b17fa700 1 -- 172.21.15.178:0/3530269065 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f05a8041000 con 0x7f05b40f5410 2023-11-15T04:59:36.109 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.103+0000 7f05b3fff700 1 --2- 172.21.15.178:0/3530269065 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f05a402ff60 0x7f05a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:36.109 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.104+0000 7f05b3fff700 1 --2- 172.21.15.178:0/3530269065 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f05a402ff60 0x7f05a4032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f05a00096a0 tx=0x7f05a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:36.110 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.109+0000 7f05b17fa700 1 -- 172.21.15.178:0/3530269065 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05a801d7f0 con 0x7f05b40f5410 2023-11-15T04:59:36.311 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.309+0000 7f05bb16c700 1 -- 172.21.15.178:0/3530269065 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f05b4045ad0 con 0x7f05b40f5410 2023-11-15T04:59:36.312 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.310+0000 7f05b17fa700 1 -- 172.21.15.178:0/3530269065 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f05a8010230 con 0x7f05b40f5410 2023-11-15T04:59:36.312 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:59:36.312 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:59:36.313 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.312+0000 7f05bb16c700 1 -- 172.21.15.178:0/3530269065 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f05a402ff60 msgr2=0x7f05a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:36.314 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.312+0000 7f05bb16c700 1 --2- 172.21.15.178:0/3530269065 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f05a402ff60 0x7f05a4032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f05a00096a0 tx=0x7f05a0006b40).stop 2023-11-15T04:59:36.314 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.312+0000 7f05bb16c700 1 -- 172.21.15.178:0/3530269065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05b40f5410 msgr2=0x7f05b4109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:36.314 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.312+0000 7f05bb16c700 1 --2- 172.21.15.178:0/3530269065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05b40f5410 0x7f05b4109450 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f05a8013510 tx=0x7f05a800b040).stop 2023-11-15T04:59:36.314 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.312+0000 7f05bb16c700 1 -- 172.21.15.178:0/3530269065 shutdown_connections 2023-11-15T04:59:36.314 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.312+0000 7f05bb16c700 1 --2- 172.21.15.178:0/3530269065 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f05a402ff60 0x7f05a4032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:36.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.312+0000 7f05bb16c700 1 --2- 172.21.15.178:0/3530269065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05b40f5410 0x7f05b4109450 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:36.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.312+0000 7f05bb16c700 1 -- 172.21.15.178:0/3530269065 >> 172.21.15.178:0/3530269065 conn(0x7f05b40f0b70 msgr2=0x7f05b40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:36.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.312+0000 7f05bb16c700 1 -- 172.21.15.178:0/3530269065 shutdown_connections 2023-11-15T04:59:36.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:36.312+0000 7f05bb16c700 1 -- 172.21.15.178:0/3530269065 wait complete. 2023-11-15T04:59:36.316 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:36.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:34.347659+0000 mgr.smithi163.qesyrx (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:37.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:37 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:36.311929+0000 mon.smithi163 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.178:0/3530269065' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:59:37.711 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:59:37.712 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:59:38.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:38 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:36.348046+0000 mgr.smithi163.qesyrx (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:38.924 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.923+0000 7f1f35fe1700 1 -- 172.21.15.178:0/1920421870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f30060280 msgr2=0x7f1f300f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:38.925 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.923+0000 7f1f35fe1700 1 --2- 172.21.15.178:0/1920421870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f30060280 0x7f1f300f90f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f1f18009fb0 tx=0x7f1f18009b30).stop 2023-11-15T04:59:38.926 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.923+0000 7f1f35fe1700 1 -- 172.21.15.178:0/1920421870 shutdown_connections 2023-11-15T04:59:38.926 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.923+0000 7f1f35fe1700 1 --2- 172.21.15.178:0/1920421870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f30060280 0x7f1f300f90f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:38.927 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.923+0000 7f1f35fe1700 1 -- 172.21.15.178:0/1920421870 >> 172.21.15.178:0/1920421870 conn(0x7f1f300f0b70 msgr2=0x7f1f300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:38.927 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.923+0000 7f1f35fe1700 1 -- 172.21.15.178:0/1920421870 shutdown_connections 2023-11-15T04:59:38.928 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.923+0000 7f1f35fe1700 1 -- 172.21.15.178:0/1920421870 wait complete. 2023-11-15T04:59:38.929 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.923+0000 7f1f35fe1700 1 Processor -- start 2023-11-15T04:59:38.930 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.923+0000 7f1f35fe1700 1 -- start start 2023-11-15T04:59:38.931 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.923+0000 7f1f35fe1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f30060280 0x7f1f301066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:38.931 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.923+0000 7f1f35fe1700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f30106bd0 con 0x7f1f30060280 2023-11-15T04:59:38.932 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.924+0000 7f1f2f7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f30060280 0x7f1f301066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:38.932 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.924+0000 7f1f2f7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f30060280 0x7f1f301066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:37358/0 (socket says 172.21.15.178:37358) 2023-11-15T04:59:38.933 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.924+0000 7f1f2f7fe700 1 -- 172.21.15.178:0/1213161296 learned_addr learned my addr 172.21.15.178:0/1213161296 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:59:38.934 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.924+0000 7f1f2f7fe700 1 -- 172.21.15.178:0/1213161296 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f18005040 con 0x7f1f30060280 2023-11-15T04:59:38.935 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.924+0000 7f1f2f7fe700 1 --2- 172.21.15.178:0/1213161296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f30060280 0x7f1f301066e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f1f18013510 tx=0x7f1f1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:38.935 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.924+0000 7f1f27fff700 1 -- 172.21.15.178:0/1213161296 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f18009320 con 0x7f1f30060280 2023-11-15T04:59:38.935 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.924+0000 7f1f35fe1700 1 -- 172.21.15.178:0/1213161296 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f30106dd0 con 0x7f1f30060280 2023-11-15T04:59:38.935 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.924+0000 7f1f27fff700 1 -- 172.21.15.178:0/1213161296 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f18013f00 con 0x7f1f30060280 2023-11-15T04:59:38.936 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.924+0000 7f1f27fff700 1 -- 172.21.15.178:0/1213161296 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f18014dc0 con 0x7f1f30060280 2023-11-15T04:59:38.936 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.924+0000 7f1f35fe1700 1 -- 172.21.15.178:0/1213161296 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f30107260 con 0x7f1f30060280 2023-11-15T04:59:38.936 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.925+0000 7f1f27fff700 1 -- 172.21.15.178:0/1213161296 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1f180095f0 con 0x7f1f30060280 2023-11-15T04:59:38.936 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.925+0000 7f1f35fe1700 1 -- 172.21.15.178:0/1213161296 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f30045ad0 con 0x7f1f30060280 2023-11-15T04:59:38.937 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.925+0000 7f1f27fff700 1 --2- 172.21.15.178:0/1213161296 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1f1c02ff60 0x7f1f1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:38.937 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.925+0000 7f1f27fff700 1 -- 172.21.15.178:0/1213161296 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1f18041040 con 0x7f1f30060280 2023-11-15T04:59:38.937 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.926+0000 7f1f2effd700 1 --2- 172.21.15.178:0/1213161296 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1f1c02ff60 0x7f1f1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:38.937 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.926+0000 7f1f2effd700 1 --2- 172.21.15.178:0/1213161296 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1f1c02ff60 0x7f1f1c032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f1f200096a0 tx=0x7f1f20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:38.938 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:38.930+0000 7f1f27fff700 1 -- 172.21.15.178:0/1213161296 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f18014420 con 0x7f1f30060280 2023-11-15T04:59:39.136 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:39.136+0000 7f1f35fe1700 1 -- 172.21.15.178:0/1213161296 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f30108110 con 0x7f1f30060280 2023-11-15T04:59:39.137 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:39.136+0000 7f1f27fff700 1 -- 172.21.15.178:0/1213161296 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1f18010230 con 0x7f1f30060280 2023-11-15T04:59:39.137 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:59:39.137 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:59:39.138 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:39.138+0000 7f1f35fe1700 1 -- 172.21.15.178:0/1213161296 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1f1c02ff60 msgr2=0x7f1f1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:39.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:39.138+0000 7f1f35fe1700 1 --2- 172.21.15.178:0/1213161296 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1f1c02ff60 0x7f1f1c032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f1f200096a0 tx=0x7f1f20006b40).stop 2023-11-15T04:59:39.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:39.138+0000 7f1f35fe1700 1 -- 172.21.15.178:0/1213161296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f30060280 msgr2=0x7f1f301066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:39.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:39.138+0000 7f1f35fe1700 1 --2- 172.21.15.178:0/1213161296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f30060280 0x7f1f301066e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f1f18013510 tx=0x7f1f1800b040).stop 2023-11-15T04:59:39.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:39.138+0000 7f1f35fe1700 1 -- 172.21.15.178:0/1213161296 shutdown_connections 2023-11-15T04:59:39.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:39.138+0000 7f1f35fe1700 1 --2- 172.21.15.178:0/1213161296 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1f1c02ff60 0x7f1f1c032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:39.140 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:39.138+0000 7f1f35fe1700 1 --2- 172.21.15.178:0/1213161296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f30060280 0x7f1f301066e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:39.140 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:39.138+0000 7f1f35fe1700 1 -- 172.21.15.178:0/1213161296 >> 172.21.15.178:0/1213161296 conn(0x7f1f300f0b70 msgr2=0x7f1f300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:39.140 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:39.138+0000 7f1f35fe1700 1 -- 172.21.15.178:0/1213161296 shutdown_connections 2023-11-15T04:59:39.140 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:39.138+0000 7f1f35fe1700 1 -- 172.21.15.178:0/1213161296 wait complete. 2023-11-15T04:59:39.141 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:40.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:40 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:38.348368+0000 mgr.smithi163.qesyrx (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:40.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:40 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:39.137137+0000 mon.smithi163 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.178:0/1213161296' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:59:40.517 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:59:40.518 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:59:41.749 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.748+0000 7f054b6da700 1 -- 172.21.15.178:0/4259834668 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05440601e0 msgr2=0x7f05440f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:41.749 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.748+0000 7f054b6da700 1 --2- 172.21.15.178:0/4259834668 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05440601e0 0x7f05440f9190 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f0534009fb0 tx=0x7f0534009b30).stop 2023-11-15T04:59:41.750 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.748+0000 7f054b6da700 1 -- 172.21.15.178:0/4259834668 shutdown_connections 2023-11-15T04:59:41.750 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.748+0000 7f054b6da700 1 --2- 172.21.15.178:0/4259834668 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05440601e0 0x7f05440f9190 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:41.750 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.748+0000 7f054b6da700 1 -- 172.21.15.178:0/4259834668 >> 172.21.15.178:0/4259834668 conn(0x7f05440f0b70 msgr2=0x7f05440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:41.750 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.748+0000 7f054b6da700 1 -- 172.21.15.178:0/4259834668 shutdown_connections 2023-11-15T04:59:41.750 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.748+0000 7f054b6da700 1 -- 172.21.15.178:0/4259834668 wait complete. 2023-11-15T04:59:41.751 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.748+0000 7f054b6da700 1 Processor -- start 2023-11-15T04:59:41.751 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.748+0000 7f054b6da700 1 -- start start 2023-11-15T04:59:41.751 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.749+0000 7f054b6da700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05440601e0 0x7f05441066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:41.751 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.749+0000 7f054b6da700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0544106bd0 con 0x7f05440601e0 2023-11-15T04:59:41.751 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.749+0000 7f0549476700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05440601e0 0x7f05441066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:41.752 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.749+0000 7f0549476700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05440601e0 0x7f05441066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:37386/0 (socket says 172.21.15.178:37386) 2023-11-15T04:59:41.752 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.749+0000 7f0549476700 1 -- 172.21.15.178:0/1598691064 learned_addr learned my addr 172.21.15.178:0/1598691064 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:59:41.753 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.749+0000 7f0549476700 1 -- 172.21.15.178:0/1598691064 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0534005040 con 0x7f05440601e0 2023-11-15T04:59:41.753 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.749+0000 7f0549476700 1 --2- 172.21.15.178:0/1598691064 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05440601e0 0x7f05441066e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f0534004dd0 tx=0x7f0534009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:41.754 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.750+0000 7f0539ffb700 1 -- 172.21.15.178:0/1598691064 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0534018070 con 0x7f05440601e0 2023-11-15T04:59:41.754 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.750+0000 7f0539ffb700 1 -- 172.21.15.178:0/1598691064 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0534013410 con 0x7f05440601e0 2023-11-15T04:59:41.754 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.750+0000 7f054b6da700 1 -- 172.21.15.178:0/1598691064 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0544106dd0 con 0x7f05440601e0 2023-11-15T04:59:41.754 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.750+0000 7f0539ffb700 1 -- 172.21.15.178:0/1598691064 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f053401c3c0 con 0x7f05440601e0 2023-11-15T04:59:41.754 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.750+0000 7f054b6da700 1 -- 172.21.15.178:0/1598691064 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0544107260 con 0x7f05440601e0 2023-11-15T04:59:41.755 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.751+0000 7f0539ffb700 1 -- 172.21.15.178:0/1598691064 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f053401d420 con 0x7f05440601e0 2023-11-15T04:59:41.755 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.751+0000 7f054b6da700 1 -- 172.21.15.178:0/1598691064 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0544045ad0 con 0x7f05440601e0 2023-11-15T04:59:41.755 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.751+0000 7f0539ffb700 1 --2- 172.21.15.178:0/1598691064 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f053002ff60 0x7f0530032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:41.755 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.751+0000 7f0539ffb700 1 -- 172.21.15.178:0/1598691064 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0534041310 con 0x7f05440601e0 2023-11-15T04:59:41.755 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.751+0000 7f0548c75700 1 --2- 172.21.15.178:0/1598691064 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f053002ff60 0x7f0530032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:41.756 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.752+0000 7f0548c75700 1 --2- 172.21.15.178:0/1598691064 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f053002ff60 0x7f0530032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f05400096a0 tx=0x7f0540006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:41.756 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.756+0000 7f0539ffb700 1 -- 172.21.15.178:0/1598691064 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f053401d820 con 0x7f05440601e0 2023-11-15T04:59:41.958 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.958+0000 7f054b6da700 1 -- 172.21.15.178:0/1598691064 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f05441080b0 con 0x7f05440601e0 2023-11-15T04:59:41.960 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.959+0000 7f0539ffb700 1 -- 172.21.15.178:0/1598691064 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f053401b370 con 0x7f05440601e0 2023-11-15T04:59:41.960 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:59:41.960 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:59:41.961 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.961+0000 7f054b6da700 1 -- 172.21.15.178:0/1598691064 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f053002ff60 msgr2=0x7f0530032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:41.962 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.961+0000 7f054b6da700 1 --2- 172.21.15.178:0/1598691064 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f053002ff60 0x7f0530032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f05400096a0 tx=0x7f0540006b40).stop 2023-11-15T04:59:41.962 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.961+0000 7f054b6da700 1 -- 172.21.15.178:0/1598691064 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05440601e0 msgr2=0x7f05441066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:41.962 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.961+0000 7f054b6da700 1 --2- 172.21.15.178:0/1598691064 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05440601e0 0x7f05441066e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f0534004dd0 tx=0x7f0534009b30).stop 2023-11-15T04:59:41.962 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.961+0000 7f054b6da700 1 -- 172.21.15.178:0/1598691064 shutdown_connections 2023-11-15T04:59:41.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.961+0000 7f054b6da700 1 --2- 172.21.15.178:0/1598691064 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f053002ff60 0x7f0530032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:41.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.961+0000 7f054b6da700 1 --2- 172.21.15.178:0/1598691064 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05440601e0 0x7f05441066e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:41.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.961+0000 7f054b6da700 1 -- 172.21.15.178:0/1598691064 >> 172.21.15.178:0/1598691064 conn(0x7f05440f0b70 msgr2=0x7f05440f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:41.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.961+0000 7f054b6da700 1 -- 172.21.15.178:0/1598691064 shutdown_connections 2023-11-15T04:59:41.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:41.961+0000 7f054b6da700 1 -- 172.21.15.178:0/1598691064 wait complete. 2023-11-15T04:59:41.964 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:42.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:42 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:40.348722+0000 mgr.smithi163.qesyrx (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:42.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:42 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:41.959739+0000 mon.smithi163 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.178:0/1598691064' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:59:43.353 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:59:43.354 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:59:44.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:44 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:42.349079+0000 mgr.smithi163.qesyrx (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:44.598 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.596+0000 7fccd4352700 1 -- 172.21.15.178:0/1400415313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcccc0f5410 msgr2=0x7fcccc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:44.598 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.596+0000 7fccd4352700 1 --2- 172.21.15.178:0/1400415313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcccc0f5410 0x7fcccc0f5830 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fccbc009fb0 tx=0x7fccbc009b30).stop 2023-11-15T04:59:44.599 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.596+0000 7fccd4352700 1 -- 172.21.15.178:0/1400415313 shutdown_connections 2023-11-15T04:59:44.599 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.596+0000 7fccd4352700 1 --2- 172.21.15.178:0/1400415313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcccc0f5410 0x7fcccc0f5830 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:44.599 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.596+0000 7fccd4352700 1 -- 172.21.15.178:0/1400415313 >> 172.21.15.178:0/1400415313 conn(0x7fcccc0f0b70 msgr2=0x7fcccc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:44.599 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.596+0000 7fccd4352700 1 -- 172.21.15.178:0/1400415313 shutdown_connections 2023-11-15T04:59:44.599 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.596+0000 7fccd4352700 1 -- 172.21.15.178:0/1400415313 wait complete. 2023-11-15T04:59:44.600 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.597+0000 7fccd4352700 1 Processor -- start 2023-11-15T04:59:44.600 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.597+0000 7fccd4352700 1 -- start start 2023-11-15T04:59:44.600 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.597+0000 7fccd4352700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcccc0f5410 0x7fcccc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:44.600 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.597+0000 7fccd4352700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc105b20 con 0x7fcccc0f5410 2023-11-15T04:59:44.601 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.597+0000 7fccd20ee700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcccc0f5410 0x7fcccc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:44.601 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.597+0000 7fccd20ee700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcccc0f5410 0x7fcccc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:43244/0 (socket says 172.21.15.178:43244) 2023-11-15T04:59:44.601 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.597+0000 7fccd20ee700 1 -- 172.21.15.178:0/2165085913 learned_addr learned my addr 172.21.15.178:0/2165085913 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:59:44.601 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.598+0000 7fccd20ee700 1 -- 172.21.15.178:0/2165085913 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccbc005040 con 0x7fcccc0f5410 2023-11-15T04:59:44.601 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.598+0000 7fccd20ee700 1 --2- 172.21.15.178:0/2165085913 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcccc0f5410 0x7fcccc105630 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fccbc013510 tx=0x7fccbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:44.602 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.598+0000 7fccc2ffd700 1 -- 172.21.15.178:0/2165085913 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccbc009320 con 0x7fcccc0f5410 2023-11-15T04:59:44.602 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.598+0000 7fccd4352700 1 -- 172.21.15.178:0/2165085913 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc105d20 con 0x7fcccc0f5410 2023-11-15T04:59:44.602 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.598+0000 7fccc2ffd700 1 -- 172.21.15.178:0/2165085913 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fccbc013f00 con 0x7fcccc0f5410 2023-11-15T04:59:44.602 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.598+0000 7fccc2ffd700 1 -- 172.21.15.178:0/2165085913 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccbc014d90 con 0x7fcccc0f5410 2023-11-15T04:59:44.602 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.598+0000 7fccd4352700 1 -- 172.21.15.178:0/2165085913 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcccc1061b0 con 0x7fcccc0f5410 2023-11-15T04:59:44.603 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.599+0000 7fccc2ffd700 1 -- 172.21.15.178:0/2165085913 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fccbc0095f0 con 0x7fcccc0f5410 2023-11-15T04:59:44.604 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.599+0000 7fccd4352700 1 -- 172.21.15.178:0/2165085913 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcccc045ad0 con 0x7fcccc0f5410 2023-11-15T04:59:44.604 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.599+0000 7fccc2ffd700 1 --2- 172.21.15.178:0/2165085913 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fccb802ff60 0x7fccb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:44.604 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.599+0000 7fccc2ffd700 1 -- 172.21.15.178:0/2165085913 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fccbc040fc0 con 0x7fcccc0f5410 2023-11-15T04:59:44.604 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.599+0000 7fccd18ed700 1 --2- 172.21.15.178:0/2165085913 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fccb802ff60 0x7fccb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:44.605 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.600+0000 7fccd18ed700 1 --2- 172.21.15.178:0/2165085913 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fccb802ff60 0x7fccb8032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fccc80096a0 tx=0x7fccc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:44.605 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.604+0000 7fccc2ffd700 1 -- 172.21.15.178:0/2165085913 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fccbc014420 con 0x7fcccc0f5410 2023-11-15T04:59:44.809 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.809+0000 7fccd4352700 1 -- 172.21.15.178:0/2165085913 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcccc0fae60 con 0x7fcccc0f5410 2023-11-15T04:59:44.810 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.810+0000 7fccc2ffd700 1 -- 172.21.15.178:0/2165085913 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fccbc010230 con 0x7fcccc0f5410 2023-11-15T04:59:44.810 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:59:44.811 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:59:44.812 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.811+0000 7fccd4352700 1 -- 172.21.15.178:0/2165085913 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fccb802ff60 msgr2=0x7fccb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:44.812 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.811+0000 7fccd4352700 1 --2- 172.21.15.178:0/2165085913 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fccb802ff60 0x7fccb8032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fccc80096a0 tx=0x7fccc8006b40).stop 2023-11-15T04:59:44.812 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.811+0000 7fccd4352700 1 -- 172.21.15.178:0/2165085913 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcccc0f5410 msgr2=0x7fcccc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:44.812 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.811+0000 7fccd4352700 1 --2- 172.21.15.178:0/2165085913 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcccc0f5410 0x7fcccc105630 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fccbc013510 tx=0x7fccbc00b040).stop 2023-11-15T04:59:44.812 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.811+0000 7fccd4352700 1 -- 172.21.15.178:0/2165085913 shutdown_connections 2023-11-15T04:59:44.813 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.811+0000 7fccd4352700 1 --2- 172.21.15.178:0/2165085913 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fccb802ff60 0x7fccb8032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:44.813 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.811+0000 7fccd4352700 1 --2- 172.21.15.178:0/2165085913 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcccc0f5410 0x7fcccc105630 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:44.813 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.811+0000 7fccd4352700 1 -- 172.21.15.178:0/2165085913 >> 172.21.15.178:0/2165085913 conn(0x7fcccc0f0b70 msgr2=0x7fcccc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:44.813 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.811+0000 7fccd4352700 1 -- 172.21.15.178:0/2165085913 shutdown_connections 2023-11-15T04:59:44.813 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:44.811+0000 7fccd4352700 1 -- 172.21.15.178:0/2165085913 wait complete. 2023-11-15T04:59:44.814 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:45.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:45 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:44.810321+0000 mon.smithi163 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.178:0/2165085913' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:59:46.204 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:59:46.204 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:59:46.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:46 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:44.349480+0000 mgr.smithi163.qesyrx (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:47.397 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.395+0000 7ff830721700 1 -- 172.21.15.178:0/3064631223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff828060280 msgr2=0x7ff8280f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:47.397 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.395+0000 7ff830721700 1 --2- 172.21.15.178:0/3064631223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff828060280 0x7ff8280f90f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7ff81c009fb0 tx=0x7ff81c009b30).stop 2023-11-15T04:59:47.397 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.395+0000 7ff830721700 1 -- 172.21.15.178:0/3064631223 shutdown_connections 2023-11-15T04:59:47.398 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.395+0000 7ff830721700 1 --2- 172.21.15.178:0/3064631223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff828060280 0x7ff8280f90f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:47.398 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.395+0000 7ff830721700 1 -- 172.21.15.178:0/3064631223 >> 172.21.15.178:0/3064631223 conn(0x7ff8280f0b70 msgr2=0x7ff8280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:47.398 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.395+0000 7ff830721700 1 -- 172.21.15.178:0/3064631223 shutdown_connections 2023-11-15T04:59:47.398 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.395+0000 7ff830721700 1 -- 172.21.15.178:0/3064631223 wait complete. 2023-11-15T04:59:47.398 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.396+0000 7ff830721700 1 Processor -- start 2023-11-15T04:59:47.399 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.396+0000 7ff830721700 1 -- start start 2023-11-15T04:59:47.399 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.396+0000 7ff830721700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff828060280 0x7ff8281066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:47.399 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.396+0000 7ff830721700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff828106bb0 con 0x7ff828060280 2023-11-15T04:59:47.399 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.396+0000 7ff82e4bd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff828060280 0x7ff8281066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:47.399 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.396+0000 7ff82e4bd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff828060280 0x7ff8281066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:43264/0 (socket says 172.21.15.178:43264) 2023-11-15T04:59:47.400 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.396+0000 7ff82e4bd700 1 -- 172.21.15.178:0/231542770 learned_addr learned my addr 172.21.15.178:0/231542770 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:59:47.400 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.396+0000 7ff82e4bd700 1 -- 172.21.15.178:0/231542770 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff81c005040 con 0x7ff828060280 2023-11-15T04:59:47.400 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.396+0000 7ff82e4bd700 1 --2- 172.21.15.178:0/231542770 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff828060280 0x7ff8281066c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7ff81c013510 tx=0x7ff81c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:47.400 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.397+0000 7ff81affd700 1 -- 172.21.15.178:0/231542770 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff81c009320 con 0x7ff828060280 2023-11-15T04:59:47.400 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.397+0000 7ff830721700 1 -- 172.21.15.178:0/231542770 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff828106db0 con 0x7ff828060280 2023-11-15T04:59:47.401 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.397+0000 7ff81affd700 1 -- 172.21.15.178:0/231542770 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff81c013f00 con 0x7ff828060280 2023-11-15T04:59:47.401 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.397+0000 7ff81affd700 1 -- 172.21.15.178:0/231542770 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff81c014dc0 con 0x7ff828060280 2023-11-15T04:59:47.401 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.397+0000 7ff830721700 1 -- 172.21.15.178:0/231542770 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff828107240 con 0x7ff828060280 2023-11-15T04:59:47.402 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.397+0000 7ff81affd700 1 -- 172.21.15.178:0/231542770 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff81c014420 con 0x7ff828060280 2023-11-15T04:59:47.402 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.397+0000 7ff830721700 1 -- 172.21.15.178:0/231542770 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff828045ad0 con 0x7ff828060280 2023-11-15T04:59:47.402 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.398+0000 7ff81affd700 1 --2- 172.21.15.178:0/231542770 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff81402ff60 0x7ff814032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:47.403 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.398+0000 7ff81affd700 1 -- 172.21.15.178:0/231542770 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff81c041050 con 0x7ff828060280 2023-11-15T04:59:47.403 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.398+0000 7ff82dcbc700 1 --2- 172.21.15.178:0/231542770 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff81402ff60 0x7ff814032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:47.403 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.398+0000 7ff82dcbc700 1 --2- 172.21.15.178:0/231542770 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff81402ff60 0x7ff814032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7ff8240096a0 tx=0x7ff824006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:47.403 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.402+0000 7ff81affd700 1 -- 172.21.15.178:0/231542770 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff81c0095f0 con 0x7ff828060280 2023-11-15T04:59:47.607 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.607+0000 7ff830721700 1 -- 172.21.15.178:0/231542770 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8281080f0 con 0x7ff828060280 2023-11-15T04:59:47.608 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.608+0000 7ff81affd700 1 -- 172.21.15.178:0/231542770 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff81c01cb10 con 0x7ff828060280 2023-11-15T04:59:47.608 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:59:47.609 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:59:47.609 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.609+0000 7ff830721700 1 -- 172.21.15.178:0/231542770 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff81402ff60 msgr2=0x7ff814032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:47.610 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.609+0000 7ff830721700 1 --2- 172.21.15.178:0/231542770 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff81402ff60 0x7ff814032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7ff8240096a0 tx=0x7ff824006b40).stop 2023-11-15T04:59:47.610 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.609+0000 7ff830721700 1 -- 172.21.15.178:0/231542770 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff828060280 msgr2=0x7ff8281066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:47.610 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.609+0000 7ff830721700 1 --2- 172.21.15.178:0/231542770 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff828060280 0x7ff8281066c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7ff81c013510 tx=0x7ff81c00b040).stop 2023-11-15T04:59:47.610 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.609+0000 7ff830721700 1 -- 172.21.15.178:0/231542770 shutdown_connections 2023-11-15T04:59:47.611 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.609+0000 7ff830721700 1 --2- 172.21.15.178:0/231542770 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff81402ff60 0x7ff814032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:47.611 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.609+0000 7ff830721700 1 --2- 172.21.15.178:0/231542770 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff828060280 0x7ff8281066c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:47.611 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.609+0000 7ff830721700 1 -- 172.21.15.178:0/231542770 >> 172.21.15.178:0/231542770 conn(0x7ff8280f0b70 msgr2=0x7ff8280f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:47.611 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.609+0000 7ff830721700 1 -- 172.21.15.178:0/231542770 shutdown_connections 2023-11-15T04:59:47.611 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:47.609+0000 7ff830721700 1 -- 172.21.15.178:0/231542770 wait complete. 2023-11-15T04:59:47.612 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:48.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:48 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:46.349803+0000 mgr.smithi163.qesyrx (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:48.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:48 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:47.608390+0000 mon.smithi163 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.178:0/231542770' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:59:49.033 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:59:49.033 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:59:50.228 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.227+0000 7f9031d9e700 1 -- 172.21.15.178:0/4029445969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f902c0f6510 msgr2=0x7f902c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:50.228 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.227+0000 7f9031d9e700 1 --2- 172.21.15.178:0/4029445969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f902c0f6510 0x7f902c0f6930 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f901c009fb0 tx=0x7f901c009b30).stop 2023-11-15T04:59:50.228 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.227+0000 7f9031d9e700 1 -- 172.21.15.178:0/4029445969 shutdown_connections 2023-11-15T04:59:50.229 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.227+0000 7f9031d9e700 1 --2- 172.21.15.178:0/4029445969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f902c0f6510 0x7f902c0f6930 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:50.229 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.227+0000 7f9031d9e700 1 -- 172.21.15.178:0/4029445969 >> 172.21.15.178:0/4029445969 conn(0x7f902c0f1c70 msgr2=0x7f902c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:50.229 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.227+0000 7f9031d9e700 1 -- 172.21.15.178:0/4029445969 shutdown_connections 2023-11-15T04:59:50.229 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.228+0000 7f9031d9e700 1 -- 172.21.15.178:0/4029445969 wait complete. 2023-11-15T04:59:50.230 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.228+0000 7f9031d9e700 1 Processor -- start 2023-11-15T04:59:50.231 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.228+0000 7f9031d9e700 1 -- start start 2023-11-15T04:59:50.232 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.228+0000 7f9031d9e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f902c0f6510 0x7f902c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:50.232 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.228+0000 7f9031d9e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902c106c20 con 0x7f902c0f6510 2023-11-15T04:59:50.232 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.228+0000 7f902b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f902c0f6510 0x7f902c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:50.232 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.228+0000 7f902b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f902c0f6510 0x7f902c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:43292/0 (socket says 172.21.15.178:43292) 2023-11-15T04:59:50.232 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.228+0000 7f902b7fe700 1 -- 172.21.15.178:0/3610898103 learned_addr learned my addr 172.21.15.178:0/3610898103 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:59:50.233 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.229+0000 7f902b7fe700 1 -- 172.21.15.178:0/3610898103 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f901c005040 con 0x7f902c0f6510 2023-11-15T04:59:50.233 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.229+0000 7f902b7fe700 1 --2- 172.21.15.178:0/3610898103 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f902c0f6510 0x7f902c106730 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f901c013510 tx=0x7f901c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:50.233 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.229+0000 7f9013fff700 1 -- 172.21.15.178:0/3610898103 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f901c009320 con 0x7f902c0f6510 2023-11-15T04:59:50.233 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.229+0000 7f9031d9e700 1 -- 172.21.15.178:0/3610898103 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f902c106e20 con 0x7f902c0f6510 2023-11-15T04:59:50.233 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.229+0000 7f9013fff700 1 -- 172.21.15.178:0/3610898103 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f901c013f00 con 0x7f902c0f6510 2023-11-15T04:59:50.234 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.229+0000 7f9013fff700 1 -- 172.21.15.178:0/3610898103 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f901c014d90 con 0x7f902c0f6510 2023-11-15T04:59:50.234 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.229+0000 7f9031d9e700 1 -- 172.21.15.178:0/3610898103 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f902c1072b0 con 0x7f902c0f6510 2023-11-15T04:59:50.234 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.230+0000 7f9013fff700 1 -- 172.21.15.178:0/3610898103 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f901c0263a0 con 0x7f902c0f6510 2023-11-15T04:59:50.234 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.230+0000 7f9013fff700 1 --2- 172.21.15.178:0/3610898103 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f901402ff60 0x7f9014032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:50.234 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.230+0000 7f9031d9e700 1 -- 172.21.15.178:0/3610898103 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f902c045ad0 con 0x7f902c0f6510 2023-11-15T04:59:50.235 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.230+0000 7f9013fff700 1 -- 172.21.15.178:0/3610898103 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f901c0412e0 con 0x7f902c0f6510 2023-11-15T04:59:50.235 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.230+0000 7f902affd700 1 --2- 172.21.15.178:0/3610898103 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f901402ff60 0x7f9014032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:50.235 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.230+0000 7f902affd700 1 --2- 172.21.15.178:0/3610898103 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f901402ff60 0x7f9014032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f90200096a0 tx=0x7f9020006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:50.235 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.235+0000 7f9013fff700 1 -- 172.21.15.178:0/3610898103 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f901c012330 con 0x7f902c0f6510 2023-11-15T04:59:50.442 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.442+0000 7f9031d9e700 1 -- 172.21.15.178:0/3610898103 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f902c0fc050 con 0x7f902c0f6510 2023-11-15T04:59:50.443 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.443+0000 7f9013fff700 1 -- 172.21.15.178:0/3610898103 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f901c03a030 con 0x7f902c0f6510 2023-11-15T04:59:50.443 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:59:50.444 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:59:50.445 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.444+0000 7f9031d9e700 1 -- 172.21.15.178:0/3610898103 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f901402ff60 msgr2=0x7f9014032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:50.445 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.444+0000 7f9031d9e700 1 --2- 172.21.15.178:0/3610898103 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f901402ff60 0x7f9014032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f90200096a0 tx=0x7f9020006b40).stop 2023-11-15T04:59:50.445 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.444+0000 7f9031d9e700 1 -- 172.21.15.178:0/3610898103 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f902c0f6510 msgr2=0x7f902c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:50.445 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.444+0000 7f9031d9e700 1 --2- 172.21.15.178:0/3610898103 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f902c0f6510 0x7f902c106730 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f901c013510 tx=0x7f901c00b040).stop 2023-11-15T04:59:50.446 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.445+0000 7f9031d9e700 1 -- 172.21.15.178:0/3610898103 shutdown_connections 2023-11-15T04:59:50.446 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.445+0000 7f9031d9e700 1 --2- 172.21.15.178:0/3610898103 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f901402ff60 0x7f9014032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:50.446 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.445+0000 7f9031d9e700 1 --2- 172.21.15.178:0/3610898103 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f902c0f6510 0x7f902c106730 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:50.446 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.445+0000 7f9031d9e700 1 -- 172.21.15.178:0/3610898103 >> 172.21.15.178:0/3610898103 conn(0x7f902c0f1c70 msgr2=0x7f902c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:50.446 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.445+0000 7f9031d9e700 1 -- 172.21.15.178:0/3610898103 shutdown_connections 2023-11-15T04:59:50.447 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:50.445+0000 7f9031d9e700 1 -- 172.21.15.178:0/3610898103 wait complete. 2023-11-15T04:59:50.447 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:50.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:50 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:48.350181+0000 mgr.smithi163.qesyrx (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:51.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:51 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:50.443540+0000 mon.smithi163 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.178:0/3610898103' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:59:51.829 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:59:51.829 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:59:52.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:52 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:50.350560+0000 mgr.smithi163.qesyrx (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:53.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.080+0000 7f1de4f9d700 1 -- 172.21.15.178:0/4101455242 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1de00f51a0 msgr2=0x7f1de00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:53.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.080+0000 7f1de4f9d700 1 --2- 172.21.15.178:0/4101455242 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1de00f51a0 0x7f1de00f55c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f1dd0009fb0 tx=0x7f1dd0009b30).stop 2023-11-15T04:59:53.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.080+0000 7f1de4f9d700 1 -- 172.21.15.178:0/4101455242 shutdown_connections 2023-11-15T04:59:53.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.080+0000 7f1de4f9d700 1 --2- 172.21.15.178:0/4101455242 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1de00f51a0 0x7f1de00f55c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:53.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.080+0000 7f1de4f9d700 1 -- 172.21.15.178:0/4101455242 >> 172.21.15.178:0/4101455242 conn(0x7f1de00f09a0 msgr2=0x7f1de00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:53.084 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.081+0000 7f1de4f9d700 1 -- 172.21.15.178:0/4101455242 shutdown_connections 2023-11-15T04:59:53.084 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.081+0000 7f1de4f9d700 1 -- 172.21.15.178:0/4101455242 wait complete. 2023-11-15T04:59:53.084 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.081+0000 7f1de4f9d700 1 Processor -- start 2023-11-15T04:59:53.084 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.081+0000 7f1de4f9d700 1 -- start start 2023-11-15T04:59:53.084 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.081+0000 7f1de4f9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1de00f51a0 0x7f1de01053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:53.084 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.081+0000 7f1de4f9d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de01058a0 con 0x7f1de00f51a0 2023-11-15T04:59:53.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.082+0000 7f1ddf7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1de00f51a0 0x7f1de01053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:53.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.082+0000 7f1ddf7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1de00f51a0 0x7f1de01053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:43306/0 (socket says 172.21.15.178:43306) 2023-11-15T04:59:53.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.082+0000 7f1ddf7fe700 1 -- 172.21.15.178:0/1143865997 learned_addr learned my addr 172.21.15.178:0/1143865997 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:59:53.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.082+0000 7f1ddf7fe700 1 -- 172.21.15.178:0/1143865997 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd0005040 con 0x7f1de00f51a0 2023-11-15T04:59:53.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.082+0000 7f1ddf7fe700 1 --2- 172.21.15.178:0/1143865997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1de00f51a0 0x7f1de01053b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f1dd0013510 tx=0x7f1dd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:53.086 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.082+0000 7f1dc7fff700 1 -- 172.21.15.178:0/1143865997 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1dd0009320 con 0x7f1de00f51a0 2023-11-15T04:59:53.086 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.082+0000 7f1dc7fff700 1 -- 172.21.15.178:0/1143865997 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1dd0013f00 con 0x7f1de00f51a0 2023-11-15T04:59:53.086 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.082+0000 7f1de4f9d700 1 -- 172.21.15.178:0/1143865997 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de0105aa0 con 0x7f1de00f51a0 2023-11-15T04:59:53.086 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.082+0000 7f1dc7fff700 1 -- 172.21.15.178:0/1143865997 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1dd0014dc0 con 0x7f1de00f51a0 2023-11-15T04:59:53.086 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.082+0000 7f1de4f9d700 1 -- 172.21.15.178:0/1143865997 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de0105f30 con 0x7f1de00f51a0 2023-11-15T04:59:53.087 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.083+0000 7f1dc7fff700 1 -- 172.21.15.178:0/1143865997 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1dd0014420 con 0x7f1de00f51a0 2023-11-15T04:59:53.088 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.083+0000 7f1de4f9d700 1 -- 172.21.15.178:0/1143865997 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de0045ad0 con 0x7f1de00f51a0 2023-11-15T04:59:53.088 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.083+0000 7f1dc7fff700 1 --2- 172.21.15.178:0/1143865997 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1dc8030000 0x7f1dc80324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:53.088 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.083+0000 7f1dc7fff700 1 -- 172.21.15.178:0/1143865997 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1dd00410b0 con 0x7f1de00f51a0 2023-11-15T04:59:53.088 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.083+0000 7f1ddeffd700 1 --2- 172.21.15.178:0/1143865997 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1dc8030000 0x7f1dc80324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:53.088 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.084+0000 7f1ddeffd700 1 --2- 172.21.15.178:0/1143865997 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1dc8030000 0x7f1dc80324c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f1dd40096a0 tx=0x7f1dd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:53.089 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.088+0000 7f1dc7fff700 1 -- 172.21.15.178:0/1143865997 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1dd00095f0 con 0x7f1de00f51a0 2023-11-15T04:59:53.293 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.293+0000 7f1de4f9d700 1 -- 172.21.15.178:0/1143865997 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1de0106de0 con 0x7f1de00f51a0 2023-11-15T04:59:53.293 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.293+0000 7f1dc7fff700 1 -- 172.21.15.178:0/1143865997 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1dd001db70 con 0x7f1de00f51a0 2023-11-15T04:59:53.294 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:59:53.294 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:59:53.295 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.294+0000 7f1de4f9d700 1 -- 172.21.15.178:0/1143865997 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1dc8030000 msgr2=0x7f1dc80324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:53.295 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.294+0000 7f1de4f9d700 1 --2- 172.21.15.178:0/1143865997 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1dc8030000 0x7f1dc80324c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f1dd40096a0 tx=0x7f1dd4006b40).stop 2023-11-15T04:59:53.295 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.294+0000 7f1de4f9d700 1 -- 172.21.15.178:0/1143865997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1de00f51a0 msgr2=0x7f1de01053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:53.296 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.294+0000 7f1de4f9d700 1 --2- 172.21.15.178:0/1143865997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1de00f51a0 0x7f1de01053b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f1dd0013510 tx=0x7f1dd000b040).stop 2023-11-15T04:59:53.296 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.295+0000 7f1de4f9d700 1 -- 172.21.15.178:0/1143865997 shutdown_connections 2023-11-15T04:59:53.296 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.295+0000 7f1de4f9d700 1 --2- 172.21.15.178:0/1143865997 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1dc8030000 0x7f1dc80324c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:53.296 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.295+0000 7f1de4f9d700 1 --2- 172.21.15.178:0/1143865997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1de00f51a0 0x7f1de01053b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:53.296 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.295+0000 7f1de4f9d700 1 -- 172.21.15.178:0/1143865997 >> 172.21.15.178:0/1143865997 conn(0x7f1de00f09a0 msgr2=0x7f1de00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:53.297 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.295+0000 7f1de4f9d700 1 -- 172.21.15.178:0/1143865997 shutdown_connections 2023-11-15T04:59:53.297 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:53.295+0000 7f1de4f9d700 1 -- 172.21.15.178:0/1143865997 wait complete. 2023-11-15T04:59:53.297 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:54.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:54 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:52.350838+0000 mgr.smithi163.qesyrx (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:54.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:54 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:53.293836+0000 mon.smithi163 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.178:0/1143865997' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:59:54.690 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:59:54.691 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:59:55.939 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.937+0000 7fb541675700 1 -- 172.21.15.178:0/2224928527 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb53c0f5410 msgr2=0x7fb53c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:55.939 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.937+0000 7fb541675700 1 --2- 172.21.15.178:0/2224928527 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb53c0f5410 0x7fb53c0f5830 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fb52c009fb0 tx=0x7fb52c009b30).stop 2023-11-15T04:59:55.940 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.937+0000 7fb541675700 1 -- 172.21.15.178:0/2224928527 shutdown_connections 2023-11-15T04:59:55.940 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.937+0000 7fb541675700 1 --2- 172.21.15.178:0/2224928527 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb53c0f5410 0x7fb53c0f5830 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:55.940 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.937+0000 7fb541675700 1 -- 172.21.15.178:0/2224928527 >> 172.21.15.178:0/2224928527 conn(0x7fb53c0f0b70 msgr2=0x7fb53c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:55.940 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.937+0000 7fb541675700 1 -- 172.21.15.178:0/2224928527 shutdown_connections 2023-11-15T04:59:55.940 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.937+0000 7fb541675700 1 -- 172.21.15.178:0/2224928527 wait complete. 2023-11-15T04:59:55.941 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.937+0000 7fb541675700 1 Processor -- start 2023-11-15T04:59:55.941 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.937+0000 7fb541675700 1 -- start start 2023-11-15T04:59:55.941 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.938+0000 7fb541675700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb53c0f5410 0x7fb53c109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:55.941 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.938+0000 7fb541675700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c060220 con 0x7fb53c0f5410 2023-11-15T04:59:55.941 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.938+0000 7fb53affd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb53c0f5410 0x7fb53c109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:55.942 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.938+0000 7fb53affd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb53c0f5410 0x7fb53c109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:59690/0 (socket says 172.21.15.178:59690) 2023-11-15T04:59:55.942 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.938+0000 7fb53affd700 1 -- 172.21.15.178:0/2610305965 learned_addr learned my addr 172.21.15.178:0/2610305965 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:59:55.942 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.938+0000 7fb53affd700 1 -- 172.21.15.178:0/2610305965 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52c005040 con 0x7fb53c0f5410 2023-11-15T04:59:55.942 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.938+0000 7fb53affd700 1 --2- 172.21.15.178:0/2610305965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb53c0f5410 0x7fb53c109420 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fb52c013510 tx=0x7fb52c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:55.942 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.938+0000 7fb5237fe700 1 -- 172.21.15.178:0/2610305965 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb52c009320 con 0x7fb53c0f5410 2023-11-15T04:59:55.943 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.938+0000 7fb541675700 1 -- 172.21.15.178:0/2610305965 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb53c060420 con 0x7fb53c0f5410 2023-11-15T04:59:55.943 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.938+0000 7fb5237fe700 1 -- 172.21.15.178:0/2610305965 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb52c013f00 con 0x7fb53c0f5410 2023-11-15T04:59:55.943 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.938+0000 7fb5237fe700 1 -- 172.21.15.178:0/2610305965 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb52c014d90 con 0x7fb53c0f5410 2023-11-15T04:59:55.944 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.939+0000 7fb541675700 1 -- 172.21.15.178:0/2610305965 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb53c109ad0 con 0x7fb53c0f5410 2023-11-15T04:59:55.945 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.939+0000 7fb5237fe700 1 -- 172.21.15.178:0/2610305965 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb52c0263a0 con 0x7fb53c0f5410 2023-11-15T04:59:55.945 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.939+0000 7fb541675700 1 -- 172.21.15.178:0/2610305965 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb53c059100 con 0x7fb53c0f5410 2023-11-15T04:59:55.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.939+0000 7fb5237fe700 1 --2- 172.21.15.178:0/2610305965 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb52402ff60 0x7fb524032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:55.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.939+0000 7fb5237fe700 1 -- 172.21.15.178:0/2610305965 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb52c0412b0 con 0x7fb53c0f5410 2023-11-15T04:59:55.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.940+0000 7fb53a7fc700 1 --2- 172.21.15.178:0/2610305965 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb52402ff60 0x7fb524032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:55.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.940+0000 7fb53a7fc700 1 --2- 172.21.15.178:0/2610305965 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb52402ff60 0x7fb524032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fb5300096a0 tx=0x7fb530006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:55.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:55.944+0000 7fb5237fe700 1 -- 172.21.15.178:0/2610305965 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb52c01cad0 con 0x7fb53c0f5410 2023-11-15T04:59:56.151 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:56.150+0000 7fb541675700 1 -- 172.21.15.178:0/2610305965 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb53c045ad0 con 0x7fb53c0f5410 2023-11-15T04:59:56.151 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:56.151+0000 7fb5237fe700 1 -- 172.21.15.178:0/2610305965 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb52c01cad0 con 0x7fb53c0f5410 2023-11-15T04:59:56.152 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:59:56.152 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:59:56.153 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:56.153+0000 7fb541675700 1 -- 172.21.15.178:0/2610305965 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb52402ff60 msgr2=0x7fb524032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:56.153 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:56.153+0000 7fb541675700 1 --2- 172.21.15.178:0/2610305965 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb52402ff60 0x7fb524032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fb5300096a0 tx=0x7fb530006b40).stop 2023-11-15T04:59:56.153 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:56.153+0000 7fb541675700 1 -- 172.21.15.178:0/2610305965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb53c0f5410 msgr2=0x7fb53c109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:56.154 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:56.153+0000 7fb541675700 1 --2- 172.21.15.178:0/2610305965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb53c0f5410 0x7fb53c109420 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fb52c013510 tx=0x7fb52c00b040).stop 2023-11-15T04:59:56.154 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:56.153+0000 7fb541675700 1 -- 172.21.15.178:0/2610305965 shutdown_connections 2023-11-15T04:59:56.154 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:56.153+0000 7fb541675700 1 --2- 172.21.15.178:0/2610305965 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb52402ff60 0x7fb524032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:56.154 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:56.153+0000 7fb541675700 1 --2- 172.21.15.178:0/2610305965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb53c0f5410 0x7fb53c109420 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:56.154 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:56.153+0000 7fb541675700 1 -- 172.21.15.178:0/2610305965 >> 172.21.15.178:0/2610305965 conn(0x7fb53c0f0b70 msgr2=0x7fb53c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:56.155 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:56.153+0000 7fb541675700 1 -- 172.21.15.178:0/2610305965 shutdown_connections 2023-11-15T04:59:56.155 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:56.153+0000 7fb541675700 1 -- 172.21.15.178:0/2610305965 wait complete. 2023-11-15T04:59:56.155 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:56.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:56 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:54.351122+0000 mgr.smithi163.qesyrx (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:57.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:57 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:56.151737+0000 mon.smithi163 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.178:0/2610305965' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T04:59:57.539 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T04:59:57.540 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T04:59:58.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:58 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:56.351501+0000 mgr.smithi163.qesyrx (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T04:59:58.761 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.759+0000 7fcca1bbb700 1 -- 172.21.15.178:0/2810869163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc9c0f51a0 msgr2=0x7fcc9c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:58.761 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.759+0000 7fcca1bbb700 1 --2- 172.21.15.178:0/2810869163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc9c0f51a0 0x7fcc9c0f55c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fcc90009fb0 tx=0x7fcc90009b30).stop 2023-11-15T04:59:58.761 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.759+0000 7fcca1bbb700 1 -- 172.21.15.178:0/2810869163 shutdown_connections 2023-11-15T04:59:58.761 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.759+0000 7fcca1bbb700 1 --2- 172.21.15.178:0/2810869163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc9c0f51a0 0x7fcc9c0f55c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:58.762 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.759+0000 7fcca1bbb700 1 -- 172.21.15.178:0/2810869163 >> 172.21.15.178:0/2810869163 conn(0x7fcc9c0f09a0 msgr2=0x7fcc9c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:58.762 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.760+0000 7fcca1bbb700 1 -- 172.21.15.178:0/2810869163 shutdown_connections 2023-11-15T04:59:58.762 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.760+0000 7fcca1bbb700 1 -- 172.21.15.178:0/2810869163 wait complete. 2023-11-15T04:59:58.762 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.760+0000 7fcca1bbb700 1 Processor -- start 2023-11-15T04:59:58.762 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.760+0000 7fcca1bbb700 1 -- start start 2023-11-15T04:59:58.763 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.760+0000 7fcca1bbb700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc9c0f51a0 0x7fcc9c1064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:58.763 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.760+0000 7fcca1bbb700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c1069b0 con 0x7fcc9c0f51a0 2023-11-15T04:59:58.763 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.761+0000 7fcca0bb9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc9c0f51a0 0x7fcc9c1064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:58.763 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.761+0000 7fcca0bb9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc9c0f51a0 0x7fcc9c1064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:59712/0 (socket says 172.21.15.178:59712) 2023-11-15T04:59:58.764 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.761+0000 7fcca0bb9700 1 -- 172.21.15.178:0/526219020 learned_addr learned my addr 172.21.15.178:0/526219020 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T04:59:58.765 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.761+0000 7fcca0bb9700 1 -- 172.21.15.178:0/526219020 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc90005040 con 0x7fcc9c0f51a0 2023-11-15T04:59:58.765 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.761+0000 7fcca0bb9700 1 --2- 172.21.15.178:0/526219020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc9c0f51a0 0x7fcc9c1064c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fcc90013510 tx=0x7fcc9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:58.765 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.761+0000 7fcc997fa700 1 -- 172.21.15.178:0/526219020 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc90009320 con 0x7fcc9c0f51a0 2023-11-15T04:59:58.766 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.761+0000 7fcca1bbb700 1 -- 172.21.15.178:0/526219020 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc9c106bb0 con 0x7fcc9c0f51a0 2023-11-15T04:59:58.766 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.761+0000 7fcc997fa700 1 -- 172.21.15.178:0/526219020 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc90013f00 con 0x7fcc9c0f51a0 2023-11-15T04:59:58.766 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.761+0000 7fcc997fa700 1 -- 172.21.15.178:0/526219020 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc90014dc0 con 0x7fcc9c0f51a0 2023-11-15T04:59:58.766 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.761+0000 7fcca1bbb700 1 -- 172.21.15.178:0/526219020 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc9c107040 con 0x7fcc9c0f51a0 2023-11-15T04:59:58.766 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.762+0000 7fcc997fa700 1 -- 172.21.15.178:0/526219020 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcc900095f0 con 0x7fcc9c0f51a0 2023-11-15T04:59:58.767 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.762+0000 7fcca1bbb700 1 -- 172.21.15.178:0/526219020 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc9c045ad0 con 0x7fcc9c0f51a0 2023-11-15T04:59:58.767 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.762+0000 7fcc997fa700 1 --2- 172.21.15.178:0/526219020 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcc8c02ff60 0x7fcc8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T04:59:58.767 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.762+0000 7fcc997fa700 1 -- 172.21.15.178:0/526219020 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcc90041040 con 0x7fcc9c0f51a0 2023-11-15T04:59:58.768 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.762+0000 7fcc9bfff700 1 --2- 172.21.15.178:0/526219020 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcc8c02ff60 0x7fcc8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T04:59:58.768 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.763+0000 7fcc9bfff700 1 --2- 172.21.15.178:0/526219020 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcc8c02ff60 0x7fcc8c032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fcc880096a0 tx=0x7fcc88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T04:59:58.768 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.767+0000 7fcc997fa700 1 -- 172.21.15.178:0/526219020 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc90014420 con 0x7fcc9c0f51a0 2023-11-15T04:59:58.970 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.969+0000 7fcca1bbb700 1 -- 172.21.15.178:0/526219020 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc9c107ef0 con 0x7fcc9c0f51a0 2023-11-15T04:59:58.970 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.970+0000 7fcc997fa700 1 -- 172.21.15.178:0/526219020 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcc90010230 con 0x7fcc9c0f51a0 2023-11-15T04:59:58.971 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T04:59:58.971 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T04:59:58.972 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.972+0000 7fcca1bbb700 1 -- 172.21.15.178:0/526219020 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcc8c02ff60 msgr2=0x7fcc8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:58.972 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.972+0000 7fcca1bbb700 1 --2- 172.21.15.178:0/526219020 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcc8c02ff60 0x7fcc8c032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fcc880096a0 tx=0x7fcc88006b40).stop 2023-11-15T04:59:58.972 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.972+0000 7fcca1bbb700 1 -- 172.21.15.178:0/526219020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc9c0f51a0 msgr2=0x7fcc9c1064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T04:59:58.973 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.972+0000 7fcca1bbb700 1 --2- 172.21.15.178:0/526219020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc9c0f51a0 0x7fcc9c1064c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fcc90013510 tx=0x7fcc9000b040).stop 2023-11-15T04:59:58.973 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.972+0000 7fcca1bbb700 1 -- 172.21.15.178:0/526219020 shutdown_connections 2023-11-15T04:59:58.973 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.972+0000 7fcca1bbb700 1 --2- 172.21.15.178:0/526219020 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcc8c02ff60 0x7fcc8c032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:58.973 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.972+0000 7fcca1bbb700 1 --2- 172.21.15.178:0/526219020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc9c0f51a0 0x7fcc9c1064c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T04:59:58.973 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.972+0000 7fcca1bbb700 1 -- 172.21.15.178:0/526219020 >> 172.21.15.178:0/526219020 conn(0x7fcc9c0f09a0 msgr2=0x7fcc9c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T04:59:58.974 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.972+0000 7fcca1bbb700 1 -- 172.21.15.178:0/526219020 shutdown_connections 2023-11-15T04:59:58.974 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T04:59:58.972+0000 7fcca1bbb700 1 -- 172.21.15.178:0/526219020 wait complete. 2023-11-15T04:59:58.974 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T04:59:59.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 04:59:59 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T04:59:58.970911+0000 mon.smithi163 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.178:0/526219020' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:00.322 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:00.322 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:00:00.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:00 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T04:59:58.351770+0000 mgr.smithi163.qesyrx (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:01.520 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.517+0000 7f09d82d4700 1 -- 172.21.15.178:0/1867044452 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09d00f6510 msgr2=0x7f09d00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:01.521 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.517+0000 7f09d82d4700 1 --2- 172.21.15.178:0/1867044452 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09d00f6510 0x7f09d00f6930 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f09c0009fb0 tx=0x7f09c0009b30).stop 2023-11-15T05:00:01.521 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.517+0000 7f09d82d4700 1 -- 172.21.15.178:0/1867044452 shutdown_connections 2023-11-15T05:00:01.521 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.517+0000 7f09d82d4700 1 --2- 172.21.15.178:0/1867044452 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09d00f6510 0x7f09d00f6930 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:01.522 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.517+0000 7f09d82d4700 1 -- 172.21.15.178:0/1867044452 >> 172.21.15.178:0/1867044452 conn(0x7f09d00f1c70 msgr2=0x7f09d00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:01.522 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.517+0000 7f09d82d4700 1 -- 172.21.15.178:0/1867044452 shutdown_connections 2023-11-15T05:00:01.522 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.517+0000 7f09d82d4700 1 -- 172.21.15.178:0/1867044452 wait complete. 2023-11-15T05:00:01.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.518+0000 7f09d82d4700 1 Processor -- start 2023-11-15T05:00:01.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.518+0000 7f09d82d4700 1 -- start start 2023-11-15T05:00:01.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.518+0000 7f09d82d4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09d00f6510 0x7f09d0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:01.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.518+0000 7f09d82d4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d0106c20 con 0x7f09d00f6510 2023-11-15T05:00:01.524 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.518+0000 7f09d6070700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09d00f6510 0x7f09d0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:01.524 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.518+0000 7f09d6070700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09d00f6510 0x7f09d0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:59740/0 (socket says 172.21.15.178:59740) 2023-11-15T05:00:01.524 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.518+0000 7f09d6070700 1 -- 172.21.15.178:0/3042279945 learned_addr learned my addr 172.21.15.178:0/3042279945 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:00:01.525 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.518+0000 7f09d6070700 1 -- 172.21.15.178:0/3042279945 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c0005040 con 0x7f09d00f6510 2023-11-15T05:00:01.525 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.518+0000 7f09d6070700 1 --2- 172.21.15.178:0/3042279945 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09d00f6510 0x7f09d0106730 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f09c0013510 tx=0x7f09c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:01.525 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.519+0000 7f09c6ffd700 1 -- 172.21.15.178:0/3042279945 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09c0009320 con 0x7f09d00f6510 2023-11-15T05:00:01.525 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.519+0000 7f09c6ffd700 1 -- 172.21.15.178:0/3042279945 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09c0013f00 con 0x7f09d00f6510 2023-11-15T05:00:01.526 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.519+0000 7f09d82d4700 1 -- 172.21.15.178:0/3042279945 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d0106e20 con 0x7f09d00f6510 2023-11-15T05:00:01.526 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.519+0000 7f09c6ffd700 1 -- 172.21.15.178:0/3042279945 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09c0014d90 con 0x7f09d00f6510 2023-11-15T05:00:01.526 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.519+0000 7f09d82d4700 1 -- 172.21.15.178:0/3042279945 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d01072b0 con 0x7f09d00f6510 2023-11-15T05:00:01.526 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.519+0000 7f09c6ffd700 1 -- 172.21.15.178:0/3042279945 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09c00095f0 con 0x7f09d00f6510 2023-11-15T05:00:01.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.520+0000 7f09d82d4700 1 -- 172.21.15.178:0/3042279945 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d0045ad0 con 0x7f09d00f6510 2023-11-15T05:00:01.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.520+0000 7f09c6ffd700 1 --2- 172.21.15.178:0/3042279945 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f09bc02ffb0 0x7f09bc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:01.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.520+0000 7f09c6ffd700 1 -- 172.21.15.178:0/3042279945 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f09c00410a0 con 0x7f09d00f6510 2023-11-15T05:00:01.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.520+0000 7f09d586f700 1 --2- 172.21.15.178:0/3042279945 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f09bc02ffb0 0x7f09bc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:01.528 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.520+0000 7f09d586f700 1 --2- 172.21.15.178:0/3042279945 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f09bc02ffb0 0x7f09bc032470 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f09cc0096a0 tx=0x7f09cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:01.528 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.525+0000 7f09c6ffd700 1 -- 172.21.15.178:0/3042279945 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09c006d2b0 con 0x7f09d00f6510 2023-11-15T05:00:01.734 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.733+0000 7f09d82d4700 1 -- 172.21.15.178:0/3042279945 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09d00fbf60 con 0x7f09d00f6510 2023-11-15T05:00:01.734 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.734+0000 7f09c6ffd700 1 -- 172.21.15.178:0/3042279945 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f09c001d7f0 con 0x7f09d00f6510 2023-11-15T05:00:01.735 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:00:01.735 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:00:01.736 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.735+0000 7f09d82d4700 1 -- 172.21.15.178:0/3042279945 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f09bc02ffb0 msgr2=0x7f09bc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:01.736 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.735+0000 7f09d82d4700 1 --2- 172.21.15.178:0/3042279945 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f09bc02ffb0 0x7f09bc032470 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f09cc0096a0 tx=0x7f09cc006b40).stop 2023-11-15T05:00:01.736 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.735+0000 7f09d82d4700 1 -- 172.21.15.178:0/3042279945 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09d00f6510 msgr2=0x7f09d0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:01.736 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.735+0000 7f09d82d4700 1 --2- 172.21.15.178:0/3042279945 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09d00f6510 0x7f09d0106730 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f09c0013510 tx=0x7f09c000b040).stop 2023-11-15T05:00:01.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.736+0000 7f09d82d4700 1 -- 172.21.15.178:0/3042279945 shutdown_connections 2023-11-15T05:00:01.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.736+0000 7f09d82d4700 1 --2- 172.21.15.178:0/3042279945 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f09bc02ffb0 0x7f09bc032470 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:01.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.736+0000 7f09d82d4700 1 --2- 172.21.15.178:0/3042279945 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09d00f6510 0x7f09d0106730 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:01.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.736+0000 7f09d82d4700 1 -- 172.21.15.178:0/3042279945 >> 172.21.15.178:0/3042279945 conn(0x7f09d00f1c70 msgr2=0x7f09d00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:01.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.736+0000 7f09d82d4700 1 -- 172.21.15.178:0/3042279945 shutdown_connections 2023-11-15T05:00:01.738 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:01.736+0000 7f09d82d4700 1 -- 172.21.15.178:0/3042279945 wait complete. 2023-11-15T05:00:01.738 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:00:02.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:02 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:00.352108+0000 mgr.smithi163.qesyrx (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:02.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:02 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:01.734558+0000 mon.smithi163 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.178:0/3042279945' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:03.141 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:03.142 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:00:04.309 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.309+0000 7f0744fdc700 1 -- 172.21.15.178:0/2523015684 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07400f51a0 msgr2=0x7f07400f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:04.310 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.309+0000 7f0744fdc700 1 --2- 172.21.15.178:0/2523015684 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07400f51a0 0x7f07400f55c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f0730009fb0 tx=0x7f0730009b30).stop 2023-11-15T05:00:04.310 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.309+0000 7f0744fdc700 1 -- 172.21.15.178:0/2523015684 shutdown_connections 2023-11-15T05:00:04.310 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.309+0000 7f0744fdc700 1 --2- 172.21.15.178:0/2523015684 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07400f51a0 0x7f07400f55c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:04.310 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.309+0000 7f0744fdc700 1 -- 172.21.15.178:0/2523015684 >> 172.21.15.178:0/2523015684 conn(0x7f07400f09a0 msgr2=0x7f07400f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:04.311 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.309+0000 7f0744fdc700 1 -- 172.21.15.178:0/2523015684 shutdown_connections 2023-11-15T05:00:04.311 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.309+0000 7f0744fdc700 1 -- 172.21.15.178:0/2523015684 wait complete. 2023-11-15T05:00:04.313 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.310+0000 7f0744fdc700 1 Processor -- start 2023-11-15T05:00:04.313 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.310+0000 7f0744fdc700 1 -- start start 2023-11-15T05:00:04.313 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.310+0000 7f0744fdc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07400f51a0 0x7f07401064d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:04.313 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.310+0000 7f0744fdc700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07401069c0 con 0x7f07400f51a0 2023-11-15T05:00:04.314 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.310+0000 7f073f7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07400f51a0 0x7f07401064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:04.314 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.310+0000 7f073f7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07400f51a0 0x7f07401064d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:56982/0 (socket says 172.21.15.178:56982) 2023-11-15T05:00:04.314 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.310+0000 7f073f7fe700 1 -- 172.21.15.178:0/3318133195 learned_addr learned my addr 172.21.15.178:0/3318133195 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:00:04.314 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.311+0000 7f073f7fe700 1 -- 172.21.15.178:0/3318133195 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0730005040 con 0x7f07400f51a0 2023-11-15T05:00:04.314 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.311+0000 7f073f7fe700 1 --2- 172.21.15.178:0/3318133195 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07400f51a0 0x7f07401064d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f0730009830 tx=0x7f073000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:04.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.311+0000 7f0727fff700 1 -- 172.21.15.178:0/3318133195 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0730009320 con 0x7f07400f51a0 2023-11-15T05:00:04.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.311+0000 7f0744fdc700 1 -- 172.21.15.178:0/3318133195 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0740106bc0 con 0x7f07400f51a0 2023-11-15T05:00:04.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.311+0000 7f0727fff700 1 -- 172.21.15.178:0/3318133195 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07300136c0 con 0x7f07400f51a0 2023-11-15T05:00:04.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.311+0000 7f0727fff700 1 -- 172.21.15.178:0/3318133195 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f073001ee90 con 0x7f07400f51a0 2023-11-15T05:00:04.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.311+0000 7f0744fdc700 1 -- 172.21.15.178:0/3318133195 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0740107050 con 0x7f07400f51a0 2023-11-15T05:00:04.315 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.312+0000 7f0727fff700 1 -- 172.21.15.178:0/3318133195 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f073001e420 con 0x7f07400f51a0 2023-11-15T05:00:04.316 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.312+0000 7f0727fff700 1 --2- 172.21.15.178:0/3318133195 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f072802ff10 0x7f07280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:04.316 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.312+0000 7f0727fff700 1 -- 172.21.15.178:0/3318133195 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0730040fa0 con 0x7f07400f51a0 2023-11-15T05:00:04.316 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.312+0000 7f0744fdc700 1 -- 172.21.15.178:0/3318133195 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0740045ad0 con 0x7f07400f51a0 2023-11-15T05:00:04.316 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.312+0000 7f073effd700 1 --2- 172.21.15.178:0/3318133195 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f072802ff10 0x7f07280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:04.316 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.313+0000 7f073effd700 1 --2- 172.21.15.178:0/3318133195 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f072802ff10 0x7f07280323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f07340096a0 tx=0x7f0734006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:04.317 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.317+0000 7f0727fff700 1 -- 172.21.15.178:0/3318133195 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f073001cb50 con 0x7f07400f51a0 2023-11-15T05:00:04.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:04 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:02.352477+0000 mgr.smithi163.qesyrx (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:04.519 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.519+0000 7f0744fdc700 1 -- 172.21.15.178:0/3318133195 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0740107da0 con 0x7f07400f51a0 2023-11-15T05:00:04.520 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.520+0000 7f0727fff700 1 -- 172.21.15.178:0/3318133195 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f073001c320 con 0x7f07400f51a0 2023-11-15T05:00:04.520 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:00:04.521 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:00:04.522 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.521+0000 7f0744fdc700 1 -- 172.21.15.178:0/3318133195 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f072802ff10 msgr2=0x7f07280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:04.522 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.521+0000 7f0744fdc700 1 --2- 172.21.15.178:0/3318133195 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f072802ff10 0x7f07280323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f07340096a0 tx=0x7f0734006b40).stop 2023-11-15T05:00:04.522 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.521+0000 7f0744fdc700 1 -- 172.21.15.178:0/3318133195 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07400f51a0 msgr2=0x7f07401064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:04.522 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.521+0000 7f0744fdc700 1 --2- 172.21.15.178:0/3318133195 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07400f51a0 0x7f07401064d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f0730009830 tx=0x7f073000b040).stop 2023-11-15T05:00:04.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.521+0000 7f0744fdc700 1 -- 172.21.15.178:0/3318133195 shutdown_connections 2023-11-15T05:00:04.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.521+0000 7f0744fdc700 1 --2- 172.21.15.178:0/3318133195 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f072802ff10 0x7f07280323d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:04.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.521+0000 7f0744fdc700 1 --2- 172.21.15.178:0/3318133195 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07400f51a0 0x7f07401064d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:04.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.521+0000 7f0744fdc700 1 -- 172.21.15.178:0/3318133195 >> 172.21.15.178:0/3318133195 conn(0x7f07400f09a0 msgr2=0x7f074005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:04.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.522+0000 7f0744fdc700 1 -- 172.21.15.178:0/3318133195 shutdown_connections 2023-11-15T05:00:04.524 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:04.522+0000 7f0744fdc700 1 -- 172.21.15.178:0/3318133195 wait complete. 2023-11-15T05:00:04.524 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:00:05.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:05 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:04.520648+0000 mon.smithi163 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.178:0/3318133195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:05.915 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:05.915 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:00:06.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:06 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:04.352790+0000 mgr.smithi163.qesyrx (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:07.131 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.130+0000 7f4256377700 1 -- 172.21.15.178:0/839846599 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42500f5410 msgr2=0x7f42500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:07.131 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.130+0000 7f4256377700 1 --2- 172.21.15.178:0/839846599 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42500f5410 0x7f42500f5830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f4240009fb0 tx=0x7f4240009b30).stop 2023-11-15T05:00:07.131 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.130+0000 7f4256377700 1 -- 172.21.15.178:0/839846599 shutdown_connections 2023-11-15T05:00:07.131 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.130+0000 7f4256377700 1 --2- 172.21.15.178:0/839846599 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42500f5410 0x7f42500f5830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:07.132 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.130+0000 7f4256377700 1 -- 172.21.15.178:0/839846599 >> 172.21.15.178:0/839846599 conn(0x7f42500f0b70 msgr2=0x7f42500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:07.132 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.130+0000 7f4256377700 1 -- 172.21.15.178:0/839846599 shutdown_connections 2023-11-15T05:00:07.132 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.130+0000 7f4256377700 1 -- 172.21.15.178:0/839846599 wait complete. 2023-11-15T05:00:07.132 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.131+0000 7f4256377700 1 Processor -- start 2023-11-15T05:00:07.134 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.131+0000 7f4256377700 1 -- start start 2023-11-15T05:00:07.135 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.131+0000 7f4256377700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42500f5410 0x7f4250105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:07.135 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.131+0000 7f4256377700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4250105b20 con 0x7f42500f5410 2023-11-15T05:00:07.135 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.131+0000 7f424ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42500f5410 0x7f4250105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:07.136 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.131+0000 7f424ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42500f5410 0x7f4250105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:56996/0 (socket says 172.21.15.178:56996) 2023-11-15T05:00:07.136 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.131+0000 7f424ffff700 1 -- 172.21.15.178:0/973504512 learned_addr learned my addr 172.21.15.178:0/973504512 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:00:07.136 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.132+0000 7f424ffff700 1 -- 172.21.15.178:0/973504512 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4240005040 con 0x7f42500f5410 2023-11-15T05:00:07.136 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.132+0000 7f424ffff700 1 --2- 172.21.15.178:0/973504512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42500f5410 0x7f4250105630 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f4240013510 tx=0x7f424000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:07.137 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.132+0000 7f424cff9700 1 -- 172.21.15.178:0/973504512 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4240009320 con 0x7f42500f5410 2023-11-15T05:00:07.137 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.132+0000 7f424cff9700 1 -- 172.21.15.178:0/973504512 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4240013f00 con 0x7f42500f5410 2023-11-15T05:00:07.137 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.132+0000 7f4256377700 1 -- 172.21.15.178:0/973504512 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4250105d20 con 0x7f42500f5410 2023-11-15T05:00:07.137 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.132+0000 7f424cff9700 1 -- 172.21.15.178:0/973504512 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4240014d90 con 0x7f42500f5410 2023-11-15T05:00:07.137 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.132+0000 7f4256377700 1 -- 172.21.15.178:0/973504512 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42501061b0 con 0x7f42500f5410 2023-11-15T05:00:07.138 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.133+0000 7f424cff9700 1 -- 172.21.15.178:0/973504512 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42400263a0 con 0x7f42500f5410 2023-11-15T05:00:07.138 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.133+0000 7f424cff9700 1 --2- 172.21.15.178:0/973504512 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f423802ff60 0x7f4238032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:07.138 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.133+0000 7f424cff9700 1 -- 172.21.15.178:0/973504512 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4240041000 con 0x7f42500f5410 2023-11-15T05:00:07.142 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.133+0000 7f4256377700 1 -- 172.21.15.178:0/973504512 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4250045ad0 con 0x7f42500f5410 2023-11-15T05:00:07.144 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.133+0000 7f424f7fe700 1 --2- 172.21.15.178:0/973504512 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f423802ff60 0x7f4238032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:07.144 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.134+0000 7f424f7fe700 1 --2- 172.21.15.178:0/973504512 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f423802ff60 0x7f4238032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f42440096a0 tx=0x7f4244006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:07.144 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.138+0000 7f424cff9700 1 -- 172.21.15.178:0/973504512 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4240012920 con 0x7f42500f5410 2023-11-15T05:00:07.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.339+0000 7f4256377700 1 -- 172.21.15.178:0/973504512 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42500fae60 con 0x7f42500f5410 2023-11-15T05:00:07.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.340+0000 7f424cff9700 1 -- 172.21.15.178:0/973504512 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f424002b020 con 0x7f42500f5410 2023-11-15T05:00:07.342 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:00:07.342 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:00:07.348 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.342+0000 7f4256377700 1 -- 172.21.15.178:0/973504512 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f423802ff60 msgr2=0x7f4238032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:07.348 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.342+0000 7f4256377700 1 --2- 172.21.15.178:0/973504512 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f423802ff60 0x7f4238032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f42440096a0 tx=0x7f4244006b40).stop 2023-11-15T05:00:07.348 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.342+0000 7f4256377700 1 -- 172.21.15.178:0/973504512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42500f5410 msgr2=0x7f4250105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:07.349 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.342+0000 7f4256377700 1 --2- 172.21.15.178:0/973504512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42500f5410 0x7f4250105630 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f4240013510 tx=0x7f424000b040).stop 2023-11-15T05:00:07.349 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.342+0000 7f4256377700 1 -- 172.21.15.178:0/973504512 shutdown_connections 2023-11-15T05:00:07.349 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.342+0000 7f4256377700 1 --2- 172.21.15.178:0/973504512 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f423802ff60 0x7f4238032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:07.349 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.342+0000 7f4256377700 1 --2- 172.21.15.178:0/973504512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42500f5410 0x7f4250105630 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:07.349 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.342+0000 7f4256377700 1 -- 172.21.15.178:0/973504512 >> 172.21.15.178:0/973504512 conn(0x7f42500f0b70 msgr2=0x7f42500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:07.349 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.342+0000 7f4256377700 1 -- 172.21.15.178:0/973504512 shutdown_connections 2023-11-15T05:00:07.350 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:07.342+0000 7f4256377700 1 -- 172.21.15.178:0/973504512 wait complete. 2023-11-15T05:00:07.350 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:00:08.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:08 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:06.353137+0000 mgr.smithi163.qesyrx (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:08.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:08 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:07.340893+0000 mon.smithi163 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.178:0/973504512' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:08.732 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:08.733 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:00:09.882 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.881+0000 7efdfd8db700 1 -- 172.21.15.178:0/1567203350 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdf8060280 msgr2=0x7efdf80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:09.883 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.881+0000 7efdfd8db700 1 --2- 172.21.15.178:0/1567203350 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdf8060280 0x7efdf80f90f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7efdec009fb0 tx=0x7efdec009b30).stop 2023-11-15T05:00:09.883 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.881+0000 7efdfd8db700 1 -- 172.21.15.178:0/1567203350 shutdown_connections 2023-11-15T05:00:09.883 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.881+0000 7efdfd8db700 1 --2- 172.21.15.178:0/1567203350 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdf8060280 0x7efdf80f90f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:09.884 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.881+0000 7efdfd8db700 1 -- 172.21.15.178:0/1567203350 >> 172.21.15.178:0/1567203350 conn(0x7efdf80f0b70 msgr2=0x7efdf80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:09.884 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.881+0000 7efdfd8db700 1 -- 172.21.15.178:0/1567203350 shutdown_connections 2023-11-15T05:00:09.884 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.881+0000 7efdfd8db700 1 -- 172.21.15.178:0/1567203350 wait complete. 2023-11-15T05:00:09.886 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.882+0000 7efdfd8db700 1 Processor -- start 2023-11-15T05:00:09.886 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.882+0000 7efdfd8db700 1 -- start start 2023-11-15T05:00:09.886 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.882+0000 7efdfd8db700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdf8060280 0x7efdf80f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:09.886 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.882+0000 7efdfd8db700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf80f4ff0 con 0x7efdf8060280 2023-11-15T05:00:09.887 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.882+0000 7efdf6ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdf8060280 0x7efdf80f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:09.887 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.882+0000 7efdf6ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdf8060280 0x7efdf80f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:57010/0 (socket says 172.21.15.178:57010) 2023-11-15T05:00:09.887 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.882+0000 7efdf6ffd700 1 -- 172.21.15.178:0/3529198076 learned_addr learned my addr 172.21.15.178:0/3529198076 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:00:09.887 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.882+0000 7efdf6ffd700 1 -- 172.21.15.178:0/3529198076 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdec005040 con 0x7efdf8060280 2023-11-15T05:00:09.888 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.883+0000 7efdf6ffd700 1 --2- 172.21.15.178:0/3529198076 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdf8060280 0x7efdf80f6b70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7efdec013510 tx=0x7efdec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:09.888 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.883+0000 7efddf7fe700 1 -- 172.21.15.178:0/3529198076 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdec009320 con 0x7efdf8060280 2023-11-15T05:00:09.888 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.883+0000 7efdfd8db700 1 -- 172.21.15.178:0/3529198076 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf80f5190 con 0x7efdf8060280 2023-11-15T05:00:09.888 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.883+0000 7efddf7fe700 1 -- 172.21.15.178:0/3529198076 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efdec013f00 con 0x7efdf8060280 2023-11-15T05:00:09.889 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.883+0000 7efdfd8db700 1 -- 172.21.15.178:0/3529198076 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf80f5620 con 0x7efdf8060280 2023-11-15T05:00:09.889 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.883+0000 7efddf7fe700 1 -- 172.21.15.178:0/3529198076 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdec014d90 con 0x7efdf8060280 2023-11-15T05:00:09.889 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.884+0000 7efddf7fe700 1 -- 172.21.15.178:0/3529198076 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efdec0095f0 con 0x7efdf8060280 2023-11-15T05:00:09.889 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.884+0000 7efdfd8db700 1 -- 172.21.15.178:0/3529198076 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf8059100 con 0x7efdf8060280 2023-11-15T05:00:09.889 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.884+0000 7efddf7fe700 1 --2- 172.21.15.178:0/3529198076 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7efde002ff60 0x7efde0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:09.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.884+0000 7efddf7fe700 1 -- 172.21.15.178:0/3529198076 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efdec041020 con 0x7efdf8060280 2023-11-15T05:00:09.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.884+0000 7efdf67fc700 1 --2- 172.21.15.178:0/3529198076 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7efde002ff60 0x7efde0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:09.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.885+0000 7efdf67fc700 1 --2- 172.21.15.178:0/3529198076 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7efde002ff60 0x7efde0032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7efde80096a0 tx=0x7efde8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:09.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:09.889+0000 7efddf7fe700 1 -- 172.21.15.178:0/3529198076 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efdec014420 con 0x7efdf8060280 2023-11-15T05:00:10.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:10.099+0000 7efdfd8db700 1 -- 172.21.15.178:0/3529198076 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efdf80fbf50 con 0x7efdf8060280 2023-11-15T05:00:10.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:10.100+0000 7efddf7fe700 1 -- 172.21.15.178:0/3529198076 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efdec012980 con 0x7efdf8060280 2023-11-15T05:00:10.102 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:00:10.102 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:00:10.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:10.101+0000 7efdfd8db700 1 -- 172.21.15.178:0/3529198076 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7efde002ff60 msgr2=0x7efde0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:10.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:10.101+0000 7efdfd8db700 1 --2- 172.21.15.178:0/3529198076 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7efde002ff60 0x7efde0032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7efde80096a0 tx=0x7efde8006b40).stop 2023-11-15T05:00:10.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:10.101+0000 7efdfd8db700 1 -- 172.21.15.178:0/3529198076 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdf8060280 msgr2=0x7efdf80f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:10.104 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:10.101+0000 7efdfd8db700 1 --2- 172.21.15.178:0/3529198076 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdf8060280 0x7efdf80f6b70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7efdec013510 tx=0x7efdec00b040).stop 2023-11-15T05:00:10.104 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:10.102+0000 7efdfd8db700 1 -- 172.21.15.178:0/3529198076 shutdown_connections 2023-11-15T05:00:10.104 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:10.102+0000 7efdfd8db700 1 --2- 172.21.15.178:0/3529198076 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7efde002ff60 0x7efde0032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:10.104 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:10.102+0000 7efdfd8db700 1 --2- 172.21.15.178:0/3529198076 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdf8060280 0x7efdf80f6b70 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:10.104 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:10.102+0000 7efdfd8db700 1 -- 172.21.15.178:0/3529198076 >> 172.21.15.178:0/3529198076 conn(0x7efdf80f0b70 msgr2=0x7efdf80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:10.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:10.102+0000 7efdfd8db700 1 -- 172.21.15.178:0/3529198076 shutdown_connections 2023-11-15T05:00:10.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:10.102+0000 7efdfd8db700 1 -- 172.21.15.178:0/3529198076 wait complete. 2023-11-15T05:00:10.105 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:00:10.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:10 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:08.353539+0000 mgr.smithi163.qesyrx (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:11.454 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:11 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:10.101708+0000 mon.smithi163 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.178:0/3529198076' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:11.494 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:11.495 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:00:12.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:12 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:10.353916+0000 mgr.smithi163.qesyrx (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:12.700 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.697+0000 7f8711b9d700 1 -- 172.21.15.178:0/4243904262 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f870c0f62a0 msgr2=0x7f870c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:12.700 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.697+0000 7f8711b9d700 1 --2- 172.21.15.178:0/4243904262 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f870c0f62a0 0x7f870c0f66c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f8700009fb0 tx=0x7f8700009b30).stop 2023-11-15T05:00:12.700 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.697+0000 7f8711b9d700 1 -- 172.21.15.178:0/4243904262 shutdown_connections 2023-11-15T05:00:12.700 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.697+0000 7f8711b9d700 1 --2- 172.21.15.178:0/4243904262 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f870c0f62a0 0x7f870c0f66c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:12.701 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.698+0000 7f8711b9d700 1 -- 172.21.15.178:0/4243904262 >> 172.21.15.178:0/4243904262 conn(0x7f870c0f1aa0 msgr2=0x7f870c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:12.701 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.698+0000 7f8711b9d700 1 -- 172.21.15.178:0/4243904262 shutdown_connections 2023-11-15T05:00:12.701 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.698+0000 7f8711b9d700 1 -- 172.21.15.178:0/4243904262 wait complete. 2023-11-15T05:00:12.701 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.698+0000 7f8711b9d700 1 Processor -- start 2023-11-15T05:00:12.702 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.698+0000 7f8711b9d700 1 -- start start 2023-11-15T05:00:12.702 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.699+0000 7f8711b9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f870c0f62a0 0x7f870c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:12.702 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.699+0000 7f8711b9d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870c1069a0 con 0x7f870c0f62a0 2023-11-15T05:00:12.704 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.699+0000 7f8710b9b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f870c0f62a0 0x7f870c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:12.704 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.699+0000 7f8710b9b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f870c0f62a0 0x7f870c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:57022/0 (socket says 172.21.15.178:57022) 2023-11-15T05:00:12.704 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.699+0000 7f8710b9b700 1 -- 172.21.15.178:0/2674332054 learned_addr learned my addr 172.21.15.178:0/2674332054 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:00:12.704 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.699+0000 7f8710b9b700 1 -- 172.21.15.178:0/2674332054 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8700005040 con 0x7f870c0f62a0 2023-11-15T05:00:12.705 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.699+0000 7f8710b9b700 1 --2- 172.21.15.178:0/2674332054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f870c0f62a0 0x7f870c1064b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f8700013510 tx=0x7f870000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:12.705 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.700+0000 7f87097fa700 1 -- 172.21.15.178:0/2674332054 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8700009320 con 0x7f870c0f62a0 2023-11-15T05:00:12.705 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.700+0000 7f87097fa700 1 -- 172.21.15.178:0/2674332054 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8700013f00 con 0x7f870c0f62a0 2023-11-15T05:00:12.705 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.700+0000 7f87097fa700 1 -- 172.21.15.178:0/2674332054 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8700014dc0 con 0x7f870c0f62a0 2023-11-15T05:00:12.706 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.700+0000 7f8711b9d700 1 -- 172.21.15.178:0/2674332054 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f870c106ba0 con 0x7f870c0f62a0 2023-11-15T05:00:12.706 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.700+0000 7f8711b9d700 1 -- 172.21.15.178:0/2674332054 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f870c107030 con 0x7f870c0f62a0 2023-11-15T05:00:12.706 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.701+0000 7f8711b9d700 1 -- 172.21.15.178:0/2674332054 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f870c045ad0 con 0x7f870c0f62a0 2023-11-15T05:00:12.706 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.701+0000 7f87097fa700 1 -- 172.21.15.178:0/2674332054 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f87000095f0 con 0x7f870c0f62a0 2023-11-15T05:00:12.706 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.701+0000 7f87097fa700 1 --2- 172.21.15.178:0/2674332054 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f86fc02ffb0 0x7f86fc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:12.707 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.701+0000 7f87097fa700 1 -- 172.21.15.178:0/2674332054 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8700040ff0 con 0x7f870c0f62a0 2023-11-15T05:00:12.707 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.701+0000 7f870bfff700 1 --2- 172.21.15.178:0/2674332054 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f86fc02ffb0 0x7f86fc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:12.707 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.702+0000 7f870bfff700 1 --2- 172.21.15.178:0/2674332054 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f86fc02ffb0 0x7f86fc032470 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f86f80096a0 tx=0x7f86f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:12.708 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.706+0000 7f87097fa700 1 -- 172.21.15.178:0/2674332054 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f870006d2b0 con 0x7f870c0f62a0 2023-11-15T05:00:12.911 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.910+0000 7f8711b9d700 1 -- 172.21.15.178:0/2674332054 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f870c107ee0 con 0x7f870c0f62a0 2023-11-15T05:00:12.912 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.911+0000 7f87097fa700 1 -- 172.21.15.178:0/2674332054 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8700014420 con 0x7f870c0f62a0 2023-11-15T05:00:12.912 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:00:12.912 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:00:12.914 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.912+0000 7f8711b9d700 1 -- 172.21.15.178:0/2674332054 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f86fc02ffb0 msgr2=0x7f86fc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:12.914 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.912+0000 7f8711b9d700 1 --2- 172.21.15.178:0/2674332054 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f86fc02ffb0 0x7f86fc032470 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f86f80096a0 tx=0x7f86f8006b40).stop 2023-11-15T05:00:12.914 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.912+0000 7f8711b9d700 1 -- 172.21.15.178:0/2674332054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f870c0f62a0 msgr2=0x7f870c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:12.914 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.912+0000 7f8711b9d700 1 --2- 172.21.15.178:0/2674332054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f870c0f62a0 0x7f870c1064b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f8700013510 tx=0x7f870000b040).stop 2023-11-15T05:00:12.914 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.912+0000 7f8711b9d700 1 -- 172.21.15.178:0/2674332054 shutdown_connections 2023-11-15T05:00:12.915 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.912+0000 7f8711b9d700 1 --2- 172.21.15.178:0/2674332054 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f86fc02ffb0 0x7f86fc032470 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:12.915 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.912+0000 7f8711b9d700 1 --2- 172.21.15.178:0/2674332054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f870c0f62a0 0x7f870c1064b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:12.915 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.912+0000 7f8711b9d700 1 -- 172.21.15.178:0/2674332054 >> 172.21.15.178:0/2674332054 conn(0x7f870c0f1aa0 msgr2=0x7f870c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:12.915 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.912+0000 7f8711b9d700 1 -- 172.21.15.178:0/2674332054 shutdown_connections 2023-11-15T05:00:12.916 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:12.912+0000 7f8711b9d700 1 -- 172.21.15.178:0/2674332054 wait complete. 2023-11-15T05:00:12.916 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:00:13.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:13 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:12.912454+0000 mon.smithi163 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.178:0/2674332054' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:14.323 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:14.324 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:00:14.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:14 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:12.354198+0000 mgr.smithi163.qesyrx (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:15.521 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.519+0000 7fae1eae1700 1 -- 172.21.15.178:0/2670828861 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae180f62a0 msgr2=0x7fae180f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:15.522 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.519+0000 7fae1eae1700 1 --2- 172.21.15.178:0/2670828861 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae180f62a0 0x7fae180f66c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fae08009fb0 tx=0x7fae08009b30).stop 2023-11-15T05:00:15.522 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.519+0000 7fae1eae1700 1 -- 172.21.15.178:0/2670828861 shutdown_connections 2023-11-15T05:00:15.522 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.519+0000 7fae1eae1700 1 --2- 172.21.15.178:0/2670828861 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae180f62a0 0x7fae180f66c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:15.522 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.519+0000 7fae1eae1700 1 -- 172.21.15.178:0/2670828861 >> 172.21.15.178:0/2670828861 conn(0x7fae180f1aa0 msgr2=0x7fae180f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:15.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.519+0000 7fae1eae1700 1 -- 172.21.15.178:0/2670828861 shutdown_connections 2023-11-15T05:00:15.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.519+0000 7fae1eae1700 1 -- 172.21.15.178:0/2670828861 wait complete. 2023-11-15T05:00:15.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.519+0000 7fae1eae1700 1 Processor -- start 2023-11-15T05:00:15.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.519+0000 7fae1eae1700 1 -- start start 2023-11-15T05:00:15.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.520+0000 7fae1eae1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae180f62a0 0x7fae181064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:15.524 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.520+0000 7fae1eae1700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae181069a0 con 0x7fae180f62a0 2023-11-15T05:00:15.524 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.520+0000 7fae1dadf700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae180f62a0 0x7fae181064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:15.524 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.520+0000 7fae1dadf700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae180f62a0 0x7fae181064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:56752/0 (socket says 172.21.15.178:56752) 2023-11-15T05:00:15.524 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.520+0000 7fae1dadf700 1 -- 172.21.15.178:0/1523894742 learned_addr learned my addr 172.21.15.178:0/1523894742 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:00:15.524 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.520+0000 7fae1dadf700 1 -- 172.21.15.178:0/1523894742 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae08005040 con 0x7fae180f62a0 2023-11-15T05:00:15.525 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.520+0000 7fae1dadf700 1 --2- 172.21.15.178:0/1523894742 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae180f62a0 0x7fae181064b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fae08009830 tx=0x7fae0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:15.525 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.520+0000 7fae0e7fc700 1 -- 172.21.15.178:0/1523894742 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae08009320 con 0x7fae180f62a0 2023-11-15T05:00:15.525 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.520+0000 7fae0e7fc700 1 -- 172.21.15.178:0/1523894742 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae080136c0 con 0x7fae180f62a0 2023-11-15T05:00:15.525 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.520+0000 7fae1eae1700 1 -- 172.21.15.178:0/1523894742 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae18106ba0 con 0x7fae180f62a0 2023-11-15T05:00:15.525 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.520+0000 7fae0e7fc700 1 -- 172.21.15.178:0/1523894742 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae0801ee90 con 0x7fae180f62a0 2023-11-15T05:00:15.526 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.520+0000 7fae1eae1700 1 -- 172.21.15.178:0/1523894742 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae18107030 con 0x7fae180f62a0 2023-11-15T05:00:15.526 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.521+0000 7fae0e7fc700 1 -- 172.21.15.178:0/1523894742 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fae0801e420 con 0x7fae180f62a0 2023-11-15T05:00:15.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.521+0000 7fae1eae1700 1 -- 172.21.15.178:0/1523894742 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae18045ad0 con 0x7fae180f62a0 2023-11-15T05:00:15.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.521+0000 7fae0e7fc700 1 --2- 172.21.15.178:0/1523894742 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fae0402ff10 0x7fae040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:15.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.521+0000 7fae0e7fc700 1 -- 172.21.15.178:0/1523894742 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fae08040f40 con 0x7fae180f62a0 2023-11-15T05:00:15.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.521+0000 7fae1d2de700 1 --2- 172.21.15.178:0/1523894742 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fae0402ff10 0x7fae040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:15.528 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.522+0000 7fae1d2de700 1 --2- 172.21.15.178:0/1523894742 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fae0402ff10 0x7fae040323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fae140096a0 tx=0x7fae14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:15.528 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.526+0000 7fae0e7fc700 1 -- 172.21.15.178:0/1523894742 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae0806d2b0 con 0x7fae180f62a0 2023-11-15T05:00:15.734 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.730+0000 7fae1eae1700 1 -- 172.21.15.178:0/1523894742 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae18107e80 con 0x7fae180f62a0 2023-11-15T05:00:15.734 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.731+0000 7fae0e7fc700 1 -- 172.21.15.178:0/1523894742 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fae080095f0 con 0x7fae180f62a0 2023-11-15T05:00:15.735 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:00:15.735 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:00:15.736 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.733+0000 7fae1eae1700 1 -- 172.21.15.178:0/1523894742 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fae0402ff10 msgr2=0x7fae040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:15.736 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.733+0000 7fae1eae1700 1 --2- 172.21.15.178:0/1523894742 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fae0402ff10 0x7fae040323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fae140096a0 tx=0x7fae14006b40).stop 2023-11-15T05:00:15.736 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.733+0000 7fae1eae1700 1 -- 172.21.15.178:0/1523894742 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae180f62a0 msgr2=0x7fae181064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:15.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.733+0000 7fae1eae1700 1 --2- 172.21.15.178:0/1523894742 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae180f62a0 0x7fae181064b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fae08009830 tx=0x7fae0800b040).stop 2023-11-15T05:00:15.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.733+0000 7fae1eae1700 1 -- 172.21.15.178:0/1523894742 shutdown_connections 2023-11-15T05:00:15.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.733+0000 7fae1eae1700 1 --2- 172.21.15.178:0/1523894742 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fae0402ff10 0x7fae040323d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:15.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.733+0000 7fae1eae1700 1 --2- 172.21.15.178:0/1523894742 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae180f62a0 0x7fae181064b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:15.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.733+0000 7fae1eae1700 1 -- 172.21.15.178:0/1523894742 >> 172.21.15.178:0/1523894742 conn(0x7fae180f1aa0 msgr2=0x7fae180f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:15.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.733+0000 7fae1eae1700 1 -- 172.21.15.178:0/1523894742 shutdown_connections 2023-11-15T05:00:15.738 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:15.733+0000 7fae1eae1700 1 -- 172.21.15.178:0/1523894742 wait complete. 2023-11-15T05:00:15.738 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:00:16.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:14.354455+0000 mgr.smithi163.qesyrx (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:16.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:15.732868+0000 mon.smithi163 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.178:0/1523894742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:17.099 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:17.100 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:00:18.304 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.302+0000 7f359e332700 1 -- 172.21.15.178:0/1867470594 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35980f6a90 msgr2=0x7f35980f6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:18.304 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.302+0000 7f359e332700 1 --2- 172.21.15.178:0/1867470594 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35980f6a90 0x7f35980f6eb0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f3588009fb0 tx=0x7f3588009b30).stop 2023-11-15T05:00:18.305 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.302+0000 7f359e332700 1 -- 172.21.15.178:0/1867470594 shutdown_connections 2023-11-15T05:00:18.305 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.302+0000 7f359e332700 1 --2- 172.21.15.178:0/1867470594 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35980f6a90 0x7f35980f6eb0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:18.305 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.302+0000 7f359e332700 1 -- 172.21.15.178:0/1867470594 >> 172.21.15.178:0/1867470594 conn(0x7f35980f02e0 msgr2=0x7f35980f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:18.305 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.302+0000 7f359e332700 1 -- 172.21.15.178:0/1867470594 shutdown_connections 2023-11-15T05:00:18.306 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.302+0000 7f359e332700 1 -- 172.21.15.178:0/1867470594 wait complete. 2023-11-15T05:00:18.306 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.303+0000 7f359e332700 1 Processor -- start 2023-11-15T05:00:18.307 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.303+0000 7f359e332700 1 -- start start 2023-11-15T05:00:18.308 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.304+0000 7f359e332700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35980f6a90 0x7f35981088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:18.308 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.304+0000 7f359e332700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3598108da0 con 0x7f35980f6a90 2023-11-15T05:00:18.308 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.304+0000 7f3597fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35980f6a90 0x7f35981088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:18.308 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.304+0000 7f3597fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35980f6a90 0x7f35981088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:56770/0 (socket says 172.21.15.178:56770) 2023-11-15T05:00:18.309 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.304+0000 7f3597fff700 1 -- 172.21.15.178:0/2819789617 learned_addr learned my addr 172.21.15.178:0/2819789617 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:00:18.309 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.304+0000 7f3597fff700 1 -- 172.21.15.178:0/2819789617 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3588005040 con 0x7f35980f6a90 2023-11-15T05:00:18.309 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.304+0000 7f3597fff700 1 --2- 172.21.15.178:0/2819789617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35980f6a90 0x7f35981088b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f3588009830 tx=0x7f358800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:18.309 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.304+0000 7f3594ff9700 1 -- 172.21.15.178:0/2819789617 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3588009320 con 0x7f35980f6a90 2023-11-15T05:00:18.309 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.304+0000 7f359e332700 1 -- 172.21.15.178:0/2819789617 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3598108fa0 con 0x7f35980f6a90 2023-11-15T05:00:18.310 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.304+0000 7f3594ff9700 1 -- 172.21.15.178:0/2819789617 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35880136c0 con 0x7f35980f6a90 2023-11-15T05:00:18.310 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.304+0000 7f3594ff9700 1 -- 172.21.15.178:0/2819789617 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f358801ee90 con 0x7f35980f6a90 2023-11-15T05:00:18.310 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.304+0000 7f359e332700 1 -- 172.21.15.178:0/2819789617 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3598109430 con 0x7f35980f6a90 2023-11-15T05:00:18.310 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.305+0000 7f3594ff9700 1 -- 172.21.15.178:0/2819789617 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3588014440 con 0x7f35980f6a90 2023-11-15T05:00:18.311 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.305+0000 7f359e332700 1 -- 172.21.15.178:0/2819789617 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3598045ad0 con 0x7f35980f6a90 2023-11-15T05:00:18.311 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.305+0000 7f3594ff9700 1 --2- 172.21.15.178:0/2819789617 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f358002ff10 0x7f35800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:18.311 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.305+0000 7f3594ff9700 1 -- 172.21.15.178:0/2819789617 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3588041200 con 0x7f35980f6a90 2023-11-15T05:00:18.311 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.306+0000 7f35977fe700 1 --2- 172.21.15.178:0/2819789617 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f358002ff10 0x7f35800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:18.311 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.306+0000 7f35977fe700 1 --2- 172.21.15.178:0/2819789617 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f358002ff10 0x7f35800323d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f358c0096a0 tx=0x7f358c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:18.312 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.310+0000 7f3594ff9700 1 -- 172.21.15.178:0/2819789617 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f358801c910 con 0x7f35980f6a90 2023-11-15T05:00:18.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:18 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:16.354783+0000 mgr.smithi163.qesyrx (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:18.515 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.514+0000 7f359e332700 1 -- 172.21.15.178:0/2819789617 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f359810a2e0 con 0x7f35980f6a90 2023-11-15T05:00:18.516 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.515+0000 7f3594ff9700 1 -- 172.21.15.178:0/2819789617 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f358801c320 con 0x7f35980f6a90 2023-11-15T05:00:18.517 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:00:18.517 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:00:18.518 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.517+0000 7f359e332700 1 -- 172.21.15.178:0/2819789617 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f358002ff10 msgr2=0x7f35800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:18.518 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.517+0000 7f359e332700 1 --2- 172.21.15.178:0/2819789617 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f358002ff10 0x7f35800323d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f358c0096a0 tx=0x7f358c006b40).stop 2023-11-15T05:00:18.519 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.517+0000 7f359e332700 1 -- 172.21.15.178:0/2819789617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35980f6a90 msgr2=0x7f35981088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:18.519 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.517+0000 7f359e332700 1 --2- 172.21.15.178:0/2819789617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35980f6a90 0x7f35981088b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f3588009830 tx=0x7f358800b040).stop 2023-11-15T05:00:18.519 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.517+0000 7f359e332700 1 -- 172.21.15.178:0/2819789617 shutdown_connections 2023-11-15T05:00:18.519 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.517+0000 7f359e332700 1 --2- 172.21.15.178:0/2819789617 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f358002ff10 0x7f35800323d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:18.519 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.517+0000 7f359e332700 1 --2- 172.21.15.178:0/2819789617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35980f6a90 0x7f35981088b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:18.519 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.517+0000 7f359e332700 1 -- 172.21.15.178:0/2819789617 >> 172.21.15.178:0/2819789617 conn(0x7f35980f02e0 msgr2=0x7f35980f0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:18.520 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.517+0000 7f359e332700 1 -- 172.21.15.178:0/2819789617 shutdown_connections 2023-11-15T05:00:18.520 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:18.517+0000 7f359e332700 1 -- 172.21.15.178:0/2819789617 wait complete. 2023-11-15T05:00:18.520 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:00:19.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:19 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:18.516722+0000 mon.smithi163 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.178:0/2819789617' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:19.936 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:19.937 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:00:20.455 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:20 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:18.355051+0000 mgr.smithi163.qesyrx (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:21.078 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.074+0000 7ff143ffc700 1 -- 172.21.15.178:0/1799516593 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff13c0f6510 msgr2=0x7ff13c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:21.078 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.074+0000 7ff143ffc700 1 --2- 172.21.15.178:0/1799516593 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff13c0f6510 0x7ff13c0f6930 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7ff130009fb0 tx=0x7ff130009b30).stop 2023-11-15T05:00:21.079 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.074+0000 7ff143ffc700 1 -- 172.21.15.178:0/1799516593 shutdown_connections 2023-11-15T05:00:21.079 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.074+0000 7ff143ffc700 1 --2- 172.21.15.178:0/1799516593 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff13c0f6510 0x7ff13c0f6930 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:21.079 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.074+0000 7ff143ffc700 1 -- 172.21.15.178:0/1799516593 >> 172.21.15.178:0/1799516593 conn(0x7ff13c0f1c70 msgr2=0x7ff13c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:21.079 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.074+0000 7ff143ffc700 1 -- 172.21.15.178:0/1799516593 shutdown_connections 2023-11-15T05:00:21.079 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.074+0000 7ff143ffc700 1 -- 172.21.15.178:0/1799516593 wait complete. 2023-11-15T05:00:21.080 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.075+0000 7ff143ffc700 1 Processor -- start 2023-11-15T05:00:21.080 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.075+0000 7ff143ffc700 1 -- start start 2023-11-15T05:00:21.081 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.075+0000 7ff143ffc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff13c0f6510 0x7ff13c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:21.081 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.075+0000 7ff143ffc700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c106c20 con 0x7ff13c0f6510 2023-11-15T05:00:21.081 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.075+0000 7ff141d98700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff13c0f6510 0x7ff13c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:21.081 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.075+0000 7ff141d98700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff13c0f6510 0x7ff13c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:56788/0 (socket says 172.21.15.178:56788) 2023-11-15T05:00:21.081 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.075+0000 7ff141d98700 1 -- 172.21.15.178:0/3206236592 learned_addr learned my addr 172.21.15.178:0/3206236592 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:00:21.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.076+0000 7ff141d98700 1 -- 172.21.15.178:0/3206236592 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff130005040 con 0x7ff13c0f6510 2023-11-15T05:00:21.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.076+0000 7ff141d98700 1 --2- 172.21.15.178:0/3206236592 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff13c0f6510 0x7ff13c106730 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7ff130013510 tx=0x7ff13000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:21.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.076+0000 7ff12e7fc700 1 -- 172.21.15.178:0/3206236592 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff130009320 con 0x7ff13c0f6510 2023-11-15T05:00:21.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.076+0000 7ff143ffc700 1 -- 172.21.15.178:0/3206236592 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff13c106e20 con 0x7ff13c0f6510 2023-11-15T05:00:21.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.076+0000 7ff12e7fc700 1 -- 172.21.15.178:0/3206236592 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff130013f00 con 0x7ff13c0f6510 2023-11-15T05:00:21.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.076+0000 7ff12e7fc700 1 -- 172.21.15.178:0/3206236592 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff130014d90 con 0x7ff13c0f6510 2023-11-15T05:00:21.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.076+0000 7ff143ffc700 1 -- 172.21.15.178:0/3206236592 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff13c1072b0 con 0x7ff13c0f6510 2023-11-15T05:00:21.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.077+0000 7ff12e7fc700 1 -- 172.21.15.178:0/3206236592 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff1300263a0 con 0x7ff13c0f6510 2023-11-15T05:00:21.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.077+0000 7ff12e7fc700 1 --2- 172.21.15.178:0/3206236592 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff12802ff60 0x7ff128032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:21.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.077+0000 7ff12e7fc700 1 -- 172.21.15.178:0/3206236592 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff130040ff0 con 0x7ff13c0f6510 2023-11-15T05:00:21.084 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.077+0000 7ff143ffc700 1 -- 172.21.15.178:0/3206236592 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff13c045ad0 con 0x7ff13c0f6510 2023-11-15T05:00:21.084 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.077+0000 7ff141597700 1 --2- 172.21.15.178:0/3206236592 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff12802ff60 0x7ff128032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:21.084 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.077+0000 7ff141597700 1 --2- 172.21.15.178:0/3206236592 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff12802ff60 0x7ff128032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7ff1380096a0 tx=0x7ff138006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:21.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.082+0000 7ff12e7fc700 1 -- 172.21.15.178:0/3206236592 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff130012920 con 0x7ff13c0f6510 2023-11-15T05:00:21.291 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.290+0000 7ff143ffc700 1 -- 172.21.15.178:0/3206236592 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff13c0fc050 con 0x7ff13c0f6510 2023-11-15T05:00:21.292 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.292+0000 7ff12e7fc700 1 -- 172.21.15.178:0/3206236592 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff13002b020 con 0x7ff13c0f6510 2023-11-15T05:00:21.292 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:00:21.293 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:00:21.294 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.292+0000 7ff143ffc700 1 -- 172.21.15.178:0/3206236592 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff12802ff60 msgr2=0x7ff128032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:21.294 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.292+0000 7ff143ffc700 1 --2- 172.21.15.178:0/3206236592 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff12802ff60 0x7ff128032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7ff1380096a0 tx=0x7ff138006b40).stop 2023-11-15T05:00:21.294 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.292+0000 7ff143ffc700 1 -- 172.21.15.178:0/3206236592 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff13c0f6510 msgr2=0x7ff13c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:21.295 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.292+0000 7ff143ffc700 1 --2- 172.21.15.178:0/3206236592 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff13c0f6510 0x7ff13c106730 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7ff130013510 tx=0x7ff13000b040).stop 2023-11-15T05:00:21.295 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.293+0000 7ff143ffc700 1 -- 172.21.15.178:0/3206236592 shutdown_connections 2023-11-15T05:00:21.295 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.293+0000 7ff143ffc700 1 --2- 172.21.15.178:0/3206236592 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff12802ff60 0x7ff128032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:21.295 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.293+0000 7ff143ffc700 1 --2- 172.21.15.178:0/3206236592 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff13c0f6510 0x7ff13c106730 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:21.295 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.293+0000 7ff143ffc700 1 -- 172.21.15.178:0/3206236592 >> 172.21.15.178:0/3206236592 conn(0x7ff13c0f1c70 msgr2=0x7ff13c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:21.296 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.293+0000 7ff143ffc700 1 -- 172.21.15.178:0/3206236592 shutdown_connections 2023-11-15T05:00:21.296 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:21.293+0000 7ff143ffc700 1 -- 172.21.15.178:0/3206236592 wait complete. 2023-11-15T05:00:21.296 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:00:22.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:22 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:20.355374+0000 mgr.smithi163.qesyrx (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:22.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:22 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:21.292429+0000 mon.smithi163 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.178:0/3206236592' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:22.670 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:22.671 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:00:23.873 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.869+0000 7fc7ccbb2700 1 -- 172.21.15.178:0/737637265 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7c80f6510 msgr2=0x7fc7c80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:23.874 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.869+0000 7fc7ccbb2700 1 --2- 172.21.15.178:0/737637265 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7c80f6510 0x7fc7c80f6930 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fc7bc009fb0 tx=0x7fc7bc009b30).stop 2023-11-15T05:00:23.874 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.869+0000 7fc7ccbb2700 1 -- 172.21.15.178:0/737637265 shutdown_connections 2023-11-15T05:00:23.875 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.869+0000 7fc7ccbb2700 1 --2- 172.21.15.178:0/737637265 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7c80f6510 0x7fc7c80f6930 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:23.875 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.869+0000 7fc7ccbb2700 1 -- 172.21.15.178:0/737637265 >> 172.21.15.178:0/737637265 conn(0x7fc7c80f1c70 msgr2=0x7fc7c80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:23.876 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.869+0000 7fc7ccbb2700 1 -- 172.21.15.178:0/737637265 shutdown_connections 2023-11-15T05:00:23.876 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.869+0000 7fc7ccbb2700 1 -- 172.21.15.178:0/737637265 wait complete. 2023-11-15T05:00:23.876 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.870+0000 7fc7ccbb2700 1 Processor -- start 2023-11-15T05:00:23.877 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.870+0000 7fc7ccbb2700 1 -- start start 2023-11-15T05:00:23.877 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.870+0000 7fc7ccbb2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7c80f6510 0x7fc7c8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:23.877 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.870+0000 7fc7ccbb2700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c8106c20 con 0x7fc7c80f6510 2023-11-15T05:00:23.877 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.870+0000 7fc7c659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7c80f6510 0x7fc7c8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:23.877 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.870+0000 7fc7c659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7c80f6510 0x7fc7c8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:56802/0 (socket says 172.21.15.178:56802) 2023-11-15T05:00:23.878 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.870+0000 7fc7c659c700 1 -- 172.21.15.178:0/1658238486 learned_addr learned my addr 172.21.15.178:0/1658238486 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:00:23.878 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.870+0000 7fc7c659c700 1 -- 172.21.15.178:0/1658238486 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7bc005040 con 0x7fc7c80f6510 2023-11-15T05:00:23.878 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.870+0000 7fc7c659c700 1 --2- 172.21.15.178:0/1658238486 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7c80f6510 0x7fc7c8106730 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fc7bc013510 tx=0x7fc7bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:23.878 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.871+0000 7fc7b6ffd700 1 -- 172.21.15.178:0/1658238486 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7bc009320 con 0x7fc7c80f6510 2023-11-15T05:00:23.878 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.871+0000 7fc7ccbb2700 1 -- 172.21.15.178:0/1658238486 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c8106e20 con 0x7fc7c80f6510 2023-11-15T05:00:23.879 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.871+0000 7fc7ccbb2700 1 -- 172.21.15.178:0/1658238486 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c81072b0 con 0x7fc7c80f6510 2023-11-15T05:00:23.879 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.871+0000 7fc7b6ffd700 1 -- 172.21.15.178:0/1658238486 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7bc013f00 con 0x7fc7c80f6510 2023-11-15T05:00:23.879 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.871+0000 7fc7b6ffd700 1 -- 172.21.15.178:0/1658238486 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7bc014d90 con 0x7fc7c80f6510 2023-11-15T05:00:23.879 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.871+0000 7fc7b6ffd700 1 -- 172.21.15.178:0/1658238486 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7bc0263a0 con 0x7fc7c80f6510 2023-11-15T05:00:23.880 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.872+0000 7fc7ccbb2700 1 -- 172.21.15.178:0/1658238486 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c8045ad0 con 0x7fc7c80f6510 2023-11-15T05:00:23.880 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.872+0000 7fc7b6ffd700 1 --2- 172.21.15.178:0/1658238486 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc7b002ffb0 0x7fc7b0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:23.880 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.872+0000 7fc7b6ffd700 1 -- 172.21.15.178:0/1658238486 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7bc041370 con 0x7fc7c80f6510 2023-11-15T05:00:23.880 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.872+0000 7fc7c5d9b700 1 --2- 172.21.15.178:0/1658238486 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc7b002ffb0 0x7fc7b0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:23.881 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.872+0000 7fc7c5d9b700 1 --2- 172.21.15.178:0/1658238486 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc7b002ffb0 0x7fc7b0032470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc7b80096a0 tx=0x7fc7b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:23.881 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:23.876+0000 7fc7b6ffd700 1 -- 172.21.15.178:0/1658238486 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7bc0126f0 con 0x7fc7c80f6510 2023-11-15T05:00:24.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:24.081+0000 7fc7ccbb2700 1 -- 172.21.15.178:0/1658238486 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7c80fbf60 con 0x7fc7c80f6510 2023-11-15T05:00:24.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:24.082+0000 7fc7b6ffd700 1 -- 172.21.15.178:0/1658238486 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc7bc012330 con 0x7fc7c80f6510 2023-11-15T05:00:24.083 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:00:24.084 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:00:24.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:24.083+0000 7fc7ccbb2700 1 -- 172.21.15.178:0/1658238486 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc7b002ffb0 msgr2=0x7fc7b0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:24.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:24.083+0000 7fc7ccbb2700 1 --2- 172.21.15.178:0/1658238486 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc7b002ffb0 0x7fc7b0032470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc7b80096a0 tx=0x7fc7b8006b40).stop 2023-11-15T05:00:24.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:24.083+0000 7fc7ccbb2700 1 -- 172.21.15.178:0/1658238486 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7c80f6510 msgr2=0x7fc7c8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:24.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:24.083+0000 7fc7ccbb2700 1 --2- 172.21.15.178:0/1658238486 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7c80f6510 0x7fc7c8106730 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fc7bc013510 tx=0x7fc7bc00b040).stop 2023-11-15T05:00:24.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:24.083+0000 7fc7ccbb2700 1 -- 172.21.15.178:0/1658238486 shutdown_connections 2023-11-15T05:00:24.086 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:24.083+0000 7fc7ccbb2700 1 --2- 172.21.15.178:0/1658238486 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc7b002ffb0 0x7fc7b0032470 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:24.086 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:24.083+0000 7fc7ccbb2700 1 --2- 172.21.15.178:0/1658238486 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7c80f6510 0x7fc7c8106730 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:24.086 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:24.083+0000 7fc7ccbb2700 1 -- 172.21.15.178:0/1658238486 >> 172.21.15.178:0/1658238486 conn(0x7fc7c80f1c70 msgr2=0x7fc7c80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:24.086 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:24.084+0000 7fc7ccbb2700 1 -- 172.21.15.178:0/1658238486 shutdown_connections 2023-11-15T05:00:24.086 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:24.084+0000 7fc7ccbb2700 1 -- 172.21.15.178:0/1658238486 wait complete. 2023-11-15T05:00:24.087 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:00:24.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:24 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:22.355632+0000 mgr.smithi163.qesyrx (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:25.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:24.083306+0000 mon.smithi163 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.178:0/1658238486' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:25.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:24.452326+0000 mon.smithi163 (mon.0) 221 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:00:25.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:24.453984+0000 mon.smithi163 (mon.0) 222 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/trash_purge_schedule"}]: dispatch 2023-11-15T05:00:25.474 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:25.475 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:00:26.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:26 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:24.355873+0000 mgr.smithi163.qesyrx (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:26.599 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.596+0000 7f991e68a700 1 -- 172.21.15.178:0/2365651248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99180f5410 msgr2=0x7f99180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:26.600 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.596+0000 7f991e68a700 1 --2- 172.21.15.178:0/2365651248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99180f5410 0x7f99180f5830 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f9908009fb0 tx=0x7f9908009b30).stop 2023-11-15T05:00:26.600 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.596+0000 7f991e68a700 1 -- 172.21.15.178:0/2365651248 shutdown_connections 2023-11-15T05:00:26.600 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.596+0000 7f991e68a700 1 --2- 172.21.15.178:0/2365651248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99180f5410 0x7f99180f5830 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:26.600 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.596+0000 7f991e68a700 1 -- 172.21.15.178:0/2365651248 >> 172.21.15.178:0/2365651248 conn(0x7f99180f0b70 msgr2=0x7f99180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:26.600 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.596+0000 7f991e68a700 1 -- 172.21.15.178:0/2365651248 shutdown_connections 2023-11-15T05:00:26.601 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.596+0000 7f991e68a700 1 -- 172.21.15.178:0/2365651248 wait complete. 2023-11-15T05:00:26.601 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.597+0000 7f991e68a700 1 Processor -- start 2023-11-15T05:00:26.601 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.597+0000 7f991e68a700 1 -- start start 2023-11-15T05:00:26.601 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.597+0000 7f991e68a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99180f5410 0x7f99181093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:26.602 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.597+0000 7f991e68a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9918060220 con 0x7f99180f5410 2023-11-15T05:00:26.602 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.597+0000 7f9917fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99180f5410 0x7f99181093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:26.602 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.597+0000 7f9917fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99180f5410 0x7f99181093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:47364/0 (socket says 172.21.15.178:47364) 2023-11-15T05:00:26.602 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.597+0000 7f9917fff700 1 -- 172.21.15.178:0/4058892782 learned_addr learned my addr 172.21.15.178:0/4058892782 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:00:26.602 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.598+0000 7f9917fff700 1 -- 172.21.15.178:0/4058892782 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9908005040 con 0x7f99180f5410 2023-11-15T05:00:26.603 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.598+0000 7f9917fff700 1 --2- 172.21.15.178:0/4058892782 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99180f5410 0x7f99181093f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f9908014000 tx=0x7f990800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:26.603 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.598+0000 7f9914ff9700 1 -- 172.21.15.178:0/4058892782 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99080095f0 con 0x7f99180f5410 2023-11-15T05:00:26.603 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.598+0000 7f991e68a700 1 -- 172.21.15.178:0/4058892782 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9918060420 con 0x7f99180f5410 2023-11-15T05:00:26.603 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.598+0000 7f9914ff9700 1 -- 172.21.15.178:0/4058892782 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9908007e70 con 0x7f99180f5410 2023-11-15T05:00:26.603 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.598+0000 7f9914ff9700 1 -- 172.21.15.178:0/4058892782 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9908014ea0 con 0x7f99180f5410 2023-11-15T05:00:26.604 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.598+0000 7f991e68a700 1 -- 172.21.15.178:0/4058892782 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9918109aa0 con 0x7f99180f5410 2023-11-15T05:00:26.605 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.599+0000 7f9914ff9700 1 -- 172.21.15.178:0/4058892782 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9908025070 con 0x7f99180f5410 2023-11-15T05:00:26.605 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.599+0000 7f991e68a700 1 -- 172.21.15.178:0/4058892782 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9918059100 con 0x7f99180f5410 2023-11-15T05:00:26.605 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.599+0000 7f9914ff9700 1 --2- 172.21.15.178:0/4058892782 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f990002ff60 0x7f9900032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:26.605 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.599+0000 7f9914ff9700 1 -- 172.21.15.178:0/4058892782 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f99080409f0 con 0x7f99180f5410 2023-11-15T05:00:26.606 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.599+0000 7f99177fe700 1 --2- 172.21.15.178:0/4058892782 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f990002ff60 0x7f9900032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:26.606 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.600+0000 7f99177fe700 1 --2- 172.21.15.178:0/4058892782 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f990002ff60 0x7f9900032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f990c0096a0 tx=0x7f990c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:26.606 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.604+0000 7f9914ff9700 1 -- 172.21.15.178:0/4058892782 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f99080079c0 con 0x7f99180f5410 2023-11-15T05:00:26.810 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.809+0000 7f991e68a700 1 -- 172.21.15.178:0/4058892782 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9918045ad0 con 0x7f99180f5410 2023-11-15T05:00:26.810 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.809+0000 7f9914ff9700 1 -- 172.21.15.178:0/4058892782 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f990800f300 con 0x7f99180f5410 2023-11-15T05:00:26.811 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:00:26.811 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:00:26.812 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.811+0000 7f991e68a700 1 -- 172.21.15.178:0/4058892782 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f990002ff60 msgr2=0x7f9900032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:26.812 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.811+0000 7f991e68a700 1 --2- 172.21.15.178:0/4058892782 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f990002ff60 0x7f9900032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f990c0096a0 tx=0x7f990c006b40).stop 2023-11-15T05:00:26.812 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.811+0000 7f991e68a700 1 -- 172.21.15.178:0/4058892782 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99180f5410 msgr2=0x7f99181093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:26.813 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.811+0000 7f991e68a700 1 --2- 172.21.15.178:0/4058892782 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99180f5410 0x7f99181093f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f9908014000 tx=0x7f990800b040).stop 2023-11-15T05:00:26.813 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.811+0000 7f991e68a700 1 -- 172.21.15.178:0/4058892782 shutdown_connections 2023-11-15T05:00:26.813 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.811+0000 7f991e68a700 1 --2- 172.21.15.178:0/4058892782 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f990002ff60 0x7f9900032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:26.813 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.811+0000 7f991e68a700 1 --2- 172.21.15.178:0/4058892782 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99180f5410 0x7f99181093f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:26.813 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.811+0000 7f991e68a700 1 -- 172.21.15.178:0/4058892782 >> 172.21.15.178:0/4058892782 conn(0x7f99180f0b70 msgr2=0x7f99180f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:26.814 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.811+0000 7f991e68a700 1 -- 172.21.15.178:0/4058892782 shutdown_connections 2023-11-15T05:00:26.814 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:26.811+0000 7f991e68a700 1 -- 172.21.15.178:0/4058892782 wait complete. 2023-11-15T05:00:26.814 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:00:28.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:27 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:26.356199+0000 mgr.smithi163.qesyrx (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:28.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:27 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:26.810852+0000 mon.smithi163 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.178:0/4058892782' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:28.250 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:28.251 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:00:29.433 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.432+0000 7f3d814b4700 1 -- 172.21.15.178:0/673280362 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d7c060280 msgr2=0x7f3d7c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:29.433 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.432+0000 7f3d814b4700 1 --2- 172.21.15.178:0/673280362 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d7c060280 0x7f3d7c0f90f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f3d6c009fb0 tx=0x7f3d6c009b30).stop 2023-11-15T05:00:29.434 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.432+0000 7f3d814b4700 1 -- 172.21.15.178:0/673280362 shutdown_connections 2023-11-15T05:00:29.434 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.432+0000 7f3d814b4700 1 --2- 172.21.15.178:0/673280362 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d7c060280 0x7f3d7c0f90f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:29.434 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.432+0000 7f3d814b4700 1 -- 172.21.15.178:0/673280362 >> 172.21.15.178:0/673280362 conn(0x7f3d7c0f0b70 msgr2=0x7f3d7c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:29.434 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.432+0000 7f3d814b4700 1 -- 172.21.15.178:0/673280362 shutdown_connections 2023-11-15T05:00:29.434 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.432+0000 7f3d814b4700 1 -- 172.21.15.178:0/673280362 wait complete. 2023-11-15T05:00:29.436 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.432+0000 7f3d814b4700 1 Processor -- start 2023-11-15T05:00:29.437 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.432+0000 7f3d814b4700 1 -- start start 2023-11-15T05:00:29.437 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.433+0000 7f3d814b4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d7c060280 0x7f3d7c1066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:29.437 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.433+0000 7f3d814b4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c106bb0 con 0x7f3d7c060280 2023-11-15T05:00:29.437 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.433+0000 7f3d7affd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d7c060280 0x7f3d7c1066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:29.437 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.433+0000 7f3d7affd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d7c060280 0x7f3d7c1066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:47390/0 (socket says 172.21.15.178:47390) 2023-11-15T05:00:29.438 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.433+0000 7f3d7affd700 1 -- 172.21.15.178:0/120582733 learned_addr learned my addr 172.21.15.178:0/120582733 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:00:29.438 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.433+0000 7f3d7affd700 1 -- 172.21.15.178:0/120582733 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6c005040 con 0x7f3d7c060280 2023-11-15T05:00:29.438 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.433+0000 7f3d7affd700 1 --2- 172.21.15.178:0/120582733 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d7c060280 0x7f3d7c1066c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f3d6c013510 tx=0x7f3d6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:29.438 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.433+0000 7f3d637fe700 1 -- 172.21.15.178:0/120582733 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d6c009320 con 0x7f3d7c060280 2023-11-15T05:00:29.439 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.433+0000 7f3d814b4700 1 -- 172.21.15.178:0/120582733 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d7c106db0 con 0x7f3d7c060280 2023-11-15T05:00:29.439 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.433+0000 7f3d637fe700 1 -- 172.21.15.178:0/120582733 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d6c013f00 con 0x7f3d7c060280 2023-11-15T05:00:29.439 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.434+0000 7f3d637fe700 1 -- 172.21.15.178:0/120582733 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d6c014dc0 con 0x7f3d7c060280 2023-11-15T05:00:29.439 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.434+0000 7f3d814b4700 1 -- 172.21.15.178:0/120582733 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7c107240 con 0x7f3d7c060280 2023-11-15T05:00:29.439 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.434+0000 7f3d637fe700 1 -- 172.21.15.178:0/120582733 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3d6c0095f0 con 0x7f3d7c060280 2023-11-15T05:00:29.440 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.434+0000 7f3d814b4700 1 -- 172.21.15.178:0/120582733 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d7c045ad0 con 0x7f3d7c060280 2023-11-15T05:00:29.440 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.434+0000 7f3d637fe700 1 --2- 172.21.15.178:0/120582733 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3d6402ff60 0x7f3d64032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:29.440 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.435+0000 7f3d637fe700 1 -- 172.21.15.178:0/120582733 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d6c0410c0 con 0x7f3d7c060280 2023-11-15T05:00:29.440 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.435+0000 7f3d7a7fc700 1 --2- 172.21.15.178:0/120582733 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3d6402ff60 0x7f3d64032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:29.440 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.435+0000 7f3d7a7fc700 1 --2- 172.21.15.178:0/120582733 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3d6402ff60 0x7f3d64032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f3d700096a0 tx=0x7f3d70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:29.441 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.440+0000 7f3d637fe700 1 -- 172.21.15.178:0/120582733 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d6c014420 con 0x7f3d7c060280 2023-11-15T05:00:29.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.645+0000 7f3d814b4700 1 -- 172.21.15.178:0/120582733 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d7c1080f0 con 0x7f3d7c060280 2023-11-15T05:00:29.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.645+0000 7f3d637fe700 1 -- 172.21.15.178:0/120582733 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3d6c010230 con 0x7f3d7c060280 2023-11-15T05:00:29.647 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:00:29.647 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:00:29.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.647+0000 7f3d814b4700 1 -- 172.21.15.178:0/120582733 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3d6402ff60 msgr2=0x7f3d64032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:29.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.647+0000 7f3d814b4700 1 --2- 172.21.15.178:0/120582733 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3d6402ff60 0x7f3d64032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f3d700096a0 tx=0x7f3d70006b40).stop 2023-11-15T05:00:29.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.647+0000 7f3d814b4700 1 -- 172.21.15.178:0/120582733 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d7c060280 msgr2=0x7f3d7c1066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:29.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.647+0000 7f3d814b4700 1 --2- 172.21.15.178:0/120582733 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d7c060280 0x7f3d7c1066c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f3d6c013510 tx=0x7f3d6c00b040).stop 2023-11-15T05:00:29.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.647+0000 7f3d814b4700 1 -- 172.21.15.178:0/120582733 shutdown_connections 2023-11-15T05:00:29.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.647+0000 7f3d814b4700 1 --2- 172.21.15.178:0/120582733 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3d6402ff60 0x7f3d64032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:29.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.647+0000 7f3d814b4700 1 --2- 172.21.15.178:0/120582733 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d7c060280 0x7f3d7c1066c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:29.650 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.647+0000 7f3d814b4700 1 -- 172.21.15.178:0/120582733 >> 172.21.15.178:0/120582733 conn(0x7f3d7c0f0b70 msgr2=0x7f3d7c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:29.650 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.647+0000 7f3d814b4700 1 -- 172.21.15.178:0/120582733 shutdown_connections 2023-11-15T05:00:29.650 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:29.647+0000 7f3d814b4700 1 -- 172.21.15.178:0/120582733 wait complete. 2023-11-15T05:00:29.651 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:00:30.205 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:29 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:28.356496+0000 mgr.smithi163.qesyrx (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:30.205 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:29 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:29.646995+0000 mon.smithi163 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.178:0/120582733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:31.055 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:31.056 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:00:32.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:31 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:30.356819+0000 mgr.smithi163.qesyrx (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:32.258 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.256+0000 7fe50b0ce700 1 -- 172.21.15.178:0/2301760349 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5040f5410 msgr2=0x7fe5040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:32.258 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.256+0000 7fe50b0ce700 1 --2- 172.21.15.178:0/2301760349 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5040f5410 0x7fe5040f5830 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fe4f8009fb0 tx=0x7fe4f8009b30).stop 2023-11-15T05:00:32.259 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.256+0000 7fe50b0ce700 1 -- 172.21.15.178:0/2301760349 shutdown_connections 2023-11-15T05:00:32.259 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.256+0000 7fe50b0ce700 1 --2- 172.21.15.178:0/2301760349 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5040f5410 0x7fe5040f5830 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:32.259 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.256+0000 7fe50b0ce700 1 -- 172.21.15.178:0/2301760349 >> 172.21.15.178:0/2301760349 conn(0x7fe5040f0b70 msgr2=0x7fe5040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:32.259 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.256+0000 7fe50b0ce700 1 -- 172.21.15.178:0/2301760349 shutdown_connections 2023-11-15T05:00:32.259 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.256+0000 7fe50b0ce700 1 -- 172.21.15.178:0/2301760349 wait complete. 2023-11-15T05:00:32.260 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.256+0000 7fe50b0ce700 1 Processor -- start 2023-11-15T05:00:32.260 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.257+0000 7fe50b0ce700 1 -- start start 2023-11-15T05:00:32.260 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.257+0000 7fe50b0ce700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5040f5410 0x7fe5040fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:32.260 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.257+0000 7fe50b0ce700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5040fb820 con 0x7fe5040f5410 2023-11-15T05:00:32.262 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.257+0000 7fe508e6a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5040f5410 0x7fe5040fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:32.262 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.257+0000 7fe508e6a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5040f5410 0x7fe5040fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:47422/0 (socket says 172.21.15.178:47422) 2023-11-15T05:00:32.263 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.257+0000 7fe508e6a700 1 -- 172.21.15.178:0/2134444368 learned_addr learned my addr 172.21.15.178:0/2134444368 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:00:32.263 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.258+0000 7fe508e6a700 1 -- 172.21.15.178:0/2134444368 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f8005040 con 0x7fe5040f5410 2023-11-15T05:00:32.263 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.258+0000 7fe508e6a700 1 --2- 172.21.15.178:0/2134444368 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5040f5410 0x7fe5040fd3a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fe4f8009f80 tx=0x7fe4f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:32.264 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.258+0000 7fe5017fa700 1 -- 172.21.15.178:0/2134444368 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4f8009320 con 0x7fe5040f5410 2023-11-15T05:00:32.264 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.258+0000 7fe5017fa700 1 -- 172.21.15.178:0/2134444368 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe4f8013ec0 con 0x7fe5040f5410 2023-11-15T05:00:32.264 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.258+0000 7fe50b0ce700 1 -- 172.21.15.178:0/2134444368 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5040fb9c0 con 0x7fe5040f5410 2023-11-15T05:00:32.264 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.258+0000 7fe5017fa700 1 -- 172.21.15.178:0/2134444368 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4f801ece0 con 0x7fe5040f5410 2023-11-15T05:00:32.265 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.258+0000 7fe50b0ce700 1 -- 172.21.15.178:0/2134444368 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5040fbe50 con 0x7fe5040f5410 2023-11-15T05:00:32.265 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.259+0000 7fe50b0ce700 1 -- 172.21.15.178:0/2134444368 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe504059100 con 0x7fe5040f5410 2023-11-15T05:00:32.265 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.259+0000 7fe5017fa700 1 -- 172.21.15.178:0/2134444368 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe4f8014420 con 0x7fe5040f5410 2023-11-15T05:00:32.265 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.259+0000 7fe5017fa700 1 --2- 172.21.15.178:0/2134444368 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe4f402ff60 0x7fe4f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:32.265 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.259+0000 7fe5017fa700 1 -- 172.21.15.178:0/2134444368 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe4f8041c60 con 0x7fe5040f5410 2023-11-15T05:00:32.265 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.259+0000 7fe503fff700 1 --2- 172.21.15.178:0/2134444368 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe4f402ff60 0x7fe4f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:32.266 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.260+0000 7fe503fff700 1 --2- 172.21.15.178:0/2134444368 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe4f402ff60 0x7fe4f4032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fe4f00096a0 tx=0x7fe4f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:32.266 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.264+0000 7fe5017fa700 1 -- 172.21.15.178:0/2134444368 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe4f801c6e0 con 0x7fe5040f5410 2023-11-15T05:00:32.473 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.470+0000 7fe50b0ce700 1 -- 172.21.15.178:0/2134444368 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe504045ad0 con 0x7fe5040f5410 2023-11-15T05:00:32.473 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.471+0000 7fe5017fa700 1 -- 172.21.15.178:0/2134444368 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe4f802c2d0 con 0x7fe5040f5410 2023-11-15T05:00:32.473 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:00:32.474 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:00:32.475 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.472+0000 7fe50b0ce700 1 -- 172.21.15.178:0/2134444368 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe4f402ff60 msgr2=0x7fe4f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:32.475 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.472+0000 7fe50b0ce700 1 --2- 172.21.15.178:0/2134444368 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe4f402ff60 0x7fe4f4032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fe4f00096a0 tx=0x7fe4f0006b40).stop 2023-11-15T05:00:32.475 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.472+0000 7fe50b0ce700 1 -- 172.21.15.178:0/2134444368 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5040f5410 msgr2=0x7fe5040fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:32.475 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.472+0000 7fe50b0ce700 1 --2- 172.21.15.178:0/2134444368 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5040f5410 0x7fe5040fd3a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fe4f8009f80 tx=0x7fe4f800b040).stop 2023-11-15T05:00:32.475 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.473+0000 7fe50b0ce700 1 -- 172.21.15.178:0/2134444368 shutdown_connections 2023-11-15T05:00:32.476 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.473+0000 7fe50b0ce700 1 --2- 172.21.15.178:0/2134444368 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe4f402ff60 0x7fe4f4032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:32.476 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.473+0000 7fe50b0ce700 1 --2- 172.21.15.178:0/2134444368 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5040f5410 0x7fe5040fd3a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:32.476 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.473+0000 7fe50b0ce700 1 -- 172.21.15.178:0/2134444368 >> 172.21.15.178:0/2134444368 conn(0x7fe5040f0b70 msgr2=0x7fe50405fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:32.476 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.473+0000 7fe50b0ce700 1 -- 172.21.15.178:0/2134444368 shutdown_connections 2023-11-15T05:00:32.476 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:32.473+0000 7fe50b0ce700 1 -- 172.21.15.178:0/2134444368 wait complete. 2023-11-15T05:00:32.477 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:00:33.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:32 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:32.472679+0000 mon.smithi163 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.178:0/2134444368' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:33.858 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:33.859 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:00:34.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:33 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:32.357160+0000 mgr.smithi163.qesyrx (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:35.030 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.030+0000 7fdb3ad01700 1 -- 172.21.15.178:0/1066319464 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb340f51a0 msgr2=0x7fdb340f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:35.031 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.030+0000 7fdb3ad01700 1 --2- 172.21.15.178:0/1066319464 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb340f51a0 0x7fdb340f55c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fdb28009fb0 tx=0x7fdb28009b30).stop 2023-11-15T05:00:35.031 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.030+0000 7fdb3ad01700 1 -- 172.21.15.178:0/1066319464 shutdown_connections 2023-11-15T05:00:35.031 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.030+0000 7fdb3ad01700 1 --2- 172.21.15.178:0/1066319464 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb340f51a0 0x7fdb340f55c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:35.031 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.030+0000 7fdb3ad01700 1 -- 172.21.15.178:0/1066319464 >> 172.21.15.178:0/1066319464 conn(0x7fdb340f09a0 msgr2=0x7fdb340f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:35.032 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.030+0000 7fdb3ad01700 1 -- 172.21.15.178:0/1066319464 shutdown_connections 2023-11-15T05:00:35.032 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.030+0000 7fdb3ad01700 1 -- 172.21.15.178:0/1066319464 wait complete. 2023-11-15T05:00:35.032 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.030+0000 7fdb3ad01700 1 Processor -- start 2023-11-15T05:00:35.032 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.030+0000 7fdb3ad01700 1 -- start start 2023-11-15T05:00:35.032 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.030+0000 7fdb3ad01700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb340f51a0 0x7fdb341064f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:35.033 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.030+0000 7fdb3ad01700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb341069e0 con 0x7fdb340f51a0 2023-11-15T05:00:35.034 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.031+0000 7fdb39cff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb340f51a0 0x7fdb341064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:35.034 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.031+0000 7fdb39cff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb340f51a0 0x7fdb341064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54274/0 (socket says 172.21.15.178:54274) 2023-11-15T05:00:35.035 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.031+0000 7fdb39cff700 1 -- 172.21.15.178:0/2868363733 learned_addr learned my addr 172.21.15.178:0/2868363733 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:00:35.035 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.031+0000 7fdb39cff700 1 -- 172.21.15.178:0/2868363733 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb28005040 con 0x7fdb340f51a0 2023-11-15T05:00:35.035 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.031+0000 7fdb39cff700 1 --2- 172.21.15.178:0/2868363733 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb340f51a0 0x7fdb341064f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fdb28013510 tx=0x7fdb2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:35.035 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.031+0000 7fdb267fc700 1 -- 172.21.15.178:0/2868363733 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb28009320 con 0x7fdb340f51a0 2023-11-15T05:00:35.035 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.031+0000 7fdb3ad01700 1 -- 172.21.15.178:0/2868363733 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb34106be0 con 0x7fdb340f51a0 2023-11-15T05:00:35.036 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.031+0000 7fdb267fc700 1 -- 172.21.15.178:0/2868363733 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb28013f00 con 0x7fdb340f51a0 2023-11-15T05:00:35.036 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.031+0000 7fdb267fc700 1 -- 172.21.15.178:0/2868363733 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb28014dc0 con 0x7fdb340f51a0 2023-11-15T05:00:35.036 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.031+0000 7fdb3ad01700 1 -- 172.21.15.178:0/2868363733 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb34107070 con 0x7fdb340f51a0 2023-11-15T05:00:35.036 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.032+0000 7fdb3ad01700 1 -- 172.21.15.178:0/2868363733 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb34045ad0 con 0x7fdb340f51a0 2023-11-15T05:00:35.036 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.032+0000 7fdb267fc700 1 -- 172.21.15.178:0/2868363733 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdb280095f0 con 0x7fdb340f51a0 2023-11-15T05:00:35.037 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.032+0000 7fdb267fc700 1 --2- 172.21.15.178:0/2868363733 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdb2002ff60 0x7fdb20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:35.037 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.032+0000 7fdb267fc700 1 -- 172.21.15.178:0/2868363733 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdb280410e0 con 0x7fdb340f51a0 2023-11-15T05:00:35.037 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.032+0000 7fdb394fe700 1 --2- 172.21.15.178:0/2868363733 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdb2002ff60 0x7fdb20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:35.037 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.033+0000 7fdb394fe700 1 --2- 172.21.15.178:0/2868363733 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdb2002ff60 0x7fdb20032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fdb300096a0 tx=0x7fdb30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:35.038 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.037+0000 7fdb267fc700 1 -- 172.21.15.178:0/2868363733 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdb28014420 con 0x7fdb340f51a0 2023-11-15T05:00:35.243 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.243+0000 7fdb3ad01700 1 -- 172.21.15.178:0/2868363733 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb34108620 con 0x7fdb340f51a0 2023-11-15T05:00:35.244 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.244+0000 7fdb267fc700 1 -- 172.21.15.178:0/2868363733 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdb2801c920 con 0x7fdb340f51a0 2023-11-15T05:00:35.244 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:00:35.245 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:00:35.246 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.245+0000 7fdb3ad01700 1 -- 172.21.15.178:0/2868363733 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdb2002ff60 msgr2=0x7fdb20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:35.246 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.245+0000 7fdb3ad01700 1 --2- 172.21.15.178:0/2868363733 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdb2002ff60 0x7fdb20032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fdb300096a0 tx=0x7fdb30006b40).stop 2023-11-15T05:00:35.246 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.245+0000 7fdb3ad01700 1 -- 172.21.15.178:0/2868363733 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb340f51a0 msgr2=0x7fdb341064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:35.246 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.245+0000 7fdb3ad01700 1 --2- 172.21.15.178:0/2868363733 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb340f51a0 0x7fdb341064f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fdb28013510 tx=0x7fdb2800b040).stop 2023-11-15T05:00:35.246 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.245+0000 7fdb3ad01700 1 -- 172.21.15.178:0/2868363733 shutdown_connections 2023-11-15T05:00:35.247 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.245+0000 7fdb3ad01700 1 --2- 172.21.15.178:0/2868363733 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdb2002ff60 0x7fdb20032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:35.247 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.245+0000 7fdb3ad01700 1 --2- 172.21.15.178:0/2868363733 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb340f51a0 0x7fdb341064f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:35.247 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.245+0000 7fdb3ad01700 1 -- 172.21.15.178:0/2868363733 >> 172.21.15.178:0/2868363733 conn(0x7fdb340f09a0 msgr2=0x7fdb3405fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:35.249 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.245+0000 7fdb3ad01700 1 -- 172.21.15.178:0/2868363733 shutdown_connections 2023-11-15T05:00:35.249 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:35.245+0000 7fdb3ad01700 1 -- 172.21.15.178:0/2868363733 wait complete. 2023-11-15T05:00:35.250 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:00:36.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:34.357511+0000 mgr.smithi163.qesyrx (mgr.14162) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:36.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:35.244295+0000 mon.smithi163 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.178:0/2868363733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:36.664 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:36.665 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:00:37.831 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.830+0000 7f78e2e99700 1 -- 172.21.15.178:0/97535911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78dc0f5410 msgr2=0x7f78dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:37.831 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.830+0000 7f78e2e99700 1 --2- 172.21.15.178:0/97535911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78dc0f5410 0x7f78dc0f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f78d0009fb0 tx=0x7f78d0009b30).stop 2023-11-15T05:00:37.832 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.830+0000 7f78e2e99700 1 -- 172.21.15.178:0/97535911 shutdown_connections 2023-11-15T05:00:37.832 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.830+0000 7f78e2e99700 1 --2- 172.21.15.178:0/97535911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78dc0f5410 0x7f78dc0f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:37.832 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.830+0000 7f78e2e99700 1 -- 172.21.15.178:0/97535911 >> 172.21.15.178:0/97535911 conn(0x7f78dc0f0b70 msgr2=0x7f78dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:37.832 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.830+0000 7f78e2e99700 1 -- 172.21.15.178:0/97535911 shutdown_connections 2023-11-15T05:00:37.832 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.830+0000 7f78e2e99700 1 -- 172.21.15.178:0/97535911 wait complete. 2023-11-15T05:00:37.833 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.831+0000 7f78e2e99700 1 Processor -- start 2023-11-15T05:00:37.833 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.831+0000 7f78e2e99700 1 -- start start 2023-11-15T05:00:37.834 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.831+0000 7f78e2e99700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78dc0f5410 0x7f78dc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:37.835 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.831+0000 7f78e2e99700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78dc105b20 con 0x7f78dc0f5410 2023-11-15T05:00:37.835 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.832+0000 7f78e0c35700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78dc0f5410 0x7f78dc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:37.835 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.832+0000 7f78e0c35700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78dc0f5410 0x7f78dc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54278/0 (socket says 172.21.15.178:54278) 2023-11-15T05:00:37.835 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.832+0000 7f78e0c35700 1 -- 172.21.15.178:0/1886063121 learned_addr learned my addr 172.21.15.178:0/1886063121 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:00:37.835 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.832+0000 7f78e0c35700 1 -- 172.21.15.178:0/1886063121 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d0005040 con 0x7f78dc0f5410 2023-11-15T05:00:37.836 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.832+0000 7f78e0c35700 1 --2- 172.21.15.178:0/1886063121 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78dc0f5410 0x7f78dc105630 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f78d0013510 tx=0x7f78d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:37.836 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.832+0000 7f78d97fa700 1 -- 172.21.15.178:0/1886063121 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78d0009320 con 0x7f78dc0f5410 2023-11-15T05:00:37.836 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.832+0000 7f78e2e99700 1 -- 172.21.15.178:0/1886063121 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78dc105d20 con 0x7f78dc0f5410 2023-11-15T05:00:37.836 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.832+0000 7f78d97fa700 1 -- 172.21.15.178:0/1886063121 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f78d0013f00 con 0x7f78dc0f5410 2023-11-15T05:00:37.837 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.832+0000 7f78d97fa700 1 -- 172.21.15.178:0/1886063121 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78d0014d90 con 0x7f78dc0f5410 2023-11-15T05:00:37.837 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.832+0000 7f78e2e99700 1 -- 172.21.15.178:0/1886063121 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78dc1061b0 con 0x7f78dc0f5410 2023-11-15T05:00:37.837 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.833+0000 7f78d97fa700 1 -- 172.21.15.178:0/1886063121 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f78d00263a0 con 0x7f78dc0f5410 2023-11-15T05:00:37.837 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.833+0000 7f78e2e99700 1 -- 172.21.15.178:0/1886063121 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78dc045ad0 con 0x7f78dc0f5410 2023-11-15T05:00:37.837 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.833+0000 7f78d97fa700 1 --2- 172.21.15.178:0/1886063121 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f78cc02ff60 0x7f78cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:37.838 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.833+0000 7f78d97fa700 1 -- 172.21.15.178:0/1886063121 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f78d0041090 con 0x7f78dc0f5410 2023-11-15T05:00:37.838 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.833+0000 7f78dbfff700 1 --2- 172.21.15.178:0/1886063121 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f78cc02ff60 0x7f78cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:37.838 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.834+0000 7f78dbfff700 1 --2- 172.21.15.178:0/1886063121 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f78cc02ff60 0x7f78cc032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f78c80096a0 tx=0x7f78c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:37.838 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:37.838+0000 7f78d97fa700 1 -- 172.21.15.178:0/1886063121 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f78d00126f0 con 0x7f78dc0f5410 2023-11-15T05:00:38.042 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:38.042+0000 7f78e2e99700 1 -- 172.21.15.178:0/1886063121 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f78dc0fae60 con 0x7f78dc0f5410 2023-11-15T05:00:38.043 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:38.043+0000 7f78d97fa700 1 -- 172.21.15.178:0/1886063121 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f78d0012330 con 0x7f78dc0f5410 2023-11-15T05:00:38.043 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:00:38.044 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:00:38.045 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:38.044+0000 7f78e2e99700 1 -- 172.21.15.178:0/1886063121 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f78cc02ff60 msgr2=0x7f78cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:38.045 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:38.044+0000 7f78e2e99700 1 --2- 172.21.15.178:0/1886063121 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f78cc02ff60 0x7f78cc032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f78c80096a0 tx=0x7f78c8006b40).stop 2023-11-15T05:00:38.045 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:38.044+0000 7f78e2e99700 1 -- 172.21.15.178:0/1886063121 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78dc0f5410 msgr2=0x7f78dc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:38.045 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:38.044+0000 7f78e2e99700 1 --2- 172.21.15.178:0/1886063121 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78dc0f5410 0x7f78dc105630 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f78d0013510 tx=0x7f78d000b040).stop 2023-11-15T05:00:38.046 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:38.044+0000 7f78e2e99700 1 -- 172.21.15.178:0/1886063121 shutdown_connections 2023-11-15T05:00:38.047 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:38.044+0000 7f78e2e99700 1 --2- 172.21.15.178:0/1886063121 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f78cc02ff60 0x7f78cc032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:38.047 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:38.044+0000 7f78e2e99700 1 --2- 172.21.15.178:0/1886063121 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78dc0f5410 0x7f78dc105630 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:38.047 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:38.044+0000 7f78e2e99700 1 -- 172.21.15.178:0/1886063121 >> 172.21.15.178:0/1886063121 conn(0x7f78dc0f0b70 msgr2=0x7f78dc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:38.047 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:38.044+0000 7f78e2e99700 1 -- 172.21.15.178:0/1886063121 shutdown_connections 2023-11-15T05:00:38.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:38.044+0000 7f78e2e99700 1 -- 172.21.15.178:0/1886063121 wait complete. 2023-11-15T05:00:38.048 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:00:38.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:37 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:36.357818+0000 mgr.smithi163.qesyrx (mgr.14162) 157 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:39.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:38 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:38.043338+0000 mon.smithi163 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.178:0/1886063121' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:39.424 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:39.425 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:00:40.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:39 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:38.358174+0000 mgr.smithi163.qesyrx (mgr.14162) 158 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:40.645 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.643+0000 7ff07ae0e700 1 -- 172.21.15.178:0/3475484576 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0740601e0 msgr2=0x7ff0740f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:40.645 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.643+0000 7ff07ae0e700 1 --2- 172.21.15.178:0/3475484576 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0740601e0 0x7ff0740f9190 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7ff068009fb0 tx=0x7ff068009b30).stop 2023-11-15T05:00:40.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.643+0000 7ff07ae0e700 1 -- 172.21.15.178:0/3475484576 shutdown_connections 2023-11-15T05:00:40.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.643+0000 7ff07ae0e700 1 --2- 172.21.15.178:0/3475484576 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0740601e0 0x7ff0740f9190 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:40.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.643+0000 7ff07ae0e700 1 -- 172.21.15.178:0/3475484576 >> 172.21.15.178:0/3475484576 conn(0x7ff0740f0b70 msgr2=0x7ff0740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:40.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.643+0000 7ff07ae0e700 1 -- 172.21.15.178:0/3475484576 shutdown_connections 2023-11-15T05:00:40.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.643+0000 7ff07ae0e700 1 -- 172.21.15.178:0/3475484576 wait complete. 2023-11-15T05:00:40.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.644+0000 7ff07ae0e700 1 Processor -- start 2023-11-15T05:00:40.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.644+0000 7ff07ae0e700 1 -- start start 2023-11-15T05:00:40.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.644+0000 7ff07ae0e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0740601e0 0x7ff0741066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:40.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.644+0000 7ff07ae0e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff074106bd0 con 0x7ff0740601e0 2023-11-15T05:00:40.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.644+0000 7ff078baa700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0740601e0 0x7ff0741066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:40.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.644+0000 7ff078baa700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0740601e0 0x7ff0741066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54292/0 (socket says 172.21.15.178:54292) 2023-11-15T05:00:40.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.644+0000 7ff078baa700 1 -- 172.21.15.178:0/641488835 learned_addr learned my addr 172.21.15.178:0/641488835 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:00:40.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.645+0000 7ff078baa700 1 -- 172.21.15.178:0/641488835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff068005040 con 0x7ff0740601e0 2023-11-15T05:00:40.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.645+0000 7ff078baa700 1 --2- 172.21.15.178:0/641488835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0740601e0 0x7ff0741066e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7ff0680079e0 tx=0x7ff068009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:40.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.645+0000 7ff0717fa700 1 -- 172.21.15.178:0/641488835 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff068018070 con 0x7ff0740601e0 2023-11-15T05:00:40.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.645+0000 7ff07ae0e700 1 -- 172.21.15.178:0/641488835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff074106dd0 con 0x7ff0740601e0 2023-11-15T05:00:40.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.645+0000 7ff0717fa700 1 -- 172.21.15.178:0/641488835 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff068013410 con 0x7ff0740601e0 2023-11-15T05:00:40.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.645+0000 7ff0717fa700 1 -- 172.21.15.178:0/641488835 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff06801c3c0 con 0x7ff0740601e0 2023-11-15T05:00:40.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.645+0000 7ff07ae0e700 1 -- 172.21.15.178:0/641488835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff074107260 con 0x7ff0740601e0 2023-11-15T05:00:40.650 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.646+0000 7ff0717fa700 1 -- 172.21.15.178:0/641488835 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff06801c630 con 0x7ff0740601e0 2023-11-15T05:00:40.650 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.646+0000 7ff07ae0e700 1 -- 172.21.15.178:0/641488835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff074045ad0 con 0x7ff0740601e0 2023-11-15T05:00:40.651 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.646+0000 7ff0717fa700 1 --2- 172.21.15.178:0/641488835 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff06402ff60 0x7ff064032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:40.651 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.646+0000 7ff0717fa700 1 -- 172.21.15.178:0/641488835 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff068041790 con 0x7ff0740601e0 2023-11-15T05:00:40.651 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.646+0000 7ff073fff700 1 --2- 172.21.15.178:0/641488835 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff06402ff60 0x7ff064032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:40.651 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.647+0000 7ff073fff700 1 --2- 172.21.15.178:0/641488835 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff06402ff60 0x7ff064032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7ff0600096a0 tx=0x7ff060006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:40.652 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.651+0000 7ff0717fa700 1 -- 172.21.15.178:0/641488835 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff06801ca30 con 0x7ff0740601e0 2023-11-15T05:00:40.857 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.857+0000 7ff07ae0e700 1 -- 172.21.15.178:0/641488835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff074108110 con 0x7ff0740601e0 2023-11-15T05:00:40.858 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.858+0000 7ff0717fa700 1 -- 172.21.15.178:0/641488835 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff06801b890 con 0x7ff0740601e0 2023-11-15T05:00:40.858 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:00:40.859 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:00:40.860 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.859+0000 7ff07ae0e700 1 -- 172.21.15.178:0/641488835 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff06402ff60 msgr2=0x7ff064032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:40.860 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.859+0000 7ff07ae0e700 1 --2- 172.21.15.178:0/641488835 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff06402ff60 0x7ff064032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7ff0600096a0 tx=0x7ff060006b40).stop 2023-11-15T05:00:40.860 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.859+0000 7ff07ae0e700 1 -- 172.21.15.178:0/641488835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0740601e0 msgr2=0x7ff0741066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:40.860 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.859+0000 7ff07ae0e700 1 --2- 172.21.15.178:0/641488835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0740601e0 0x7ff0741066e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7ff0680079e0 tx=0x7ff068009b30).stop 2023-11-15T05:00:40.860 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.859+0000 7ff07ae0e700 1 -- 172.21.15.178:0/641488835 shutdown_connections 2023-11-15T05:00:40.861 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.859+0000 7ff07ae0e700 1 --2- 172.21.15.178:0/641488835 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff06402ff60 0x7ff064032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:40.861 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.859+0000 7ff07ae0e700 1 --2- 172.21.15.178:0/641488835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0740601e0 0x7ff0741066e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:40.861 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.859+0000 7ff07ae0e700 1 -- 172.21.15.178:0/641488835 >> 172.21.15.178:0/641488835 conn(0x7ff0740f0b70 msgr2=0x7ff0740f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:40.861 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.860+0000 7ff07ae0e700 1 -- 172.21.15.178:0/641488835 shutdown_connections 2023-11-15T05:00:40.861 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:40.860+0000 7ff07ae0e700 1 -- 172.21.15.178:0/641488835 wait complete. 2023-11-15T05:00:40.862 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:00:42.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:41 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:40.358567+0000 mgr.smithi163.qesyrx (mgr.14162) 159 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:42.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:41 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:40.858507+0000 mon.smithi163 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.178:0/641488835' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:42.259 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:42.260 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:00:43.430 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.429+0000 7f0737fff700 1 -- 172.21.15.178:0/4063991205 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07380601e0 msgr2=0x7f07380f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:43.431 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.429+0000 7f0737fff700 1 --2- 172.21.15.178:0/4063991205 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07380601e0 0x7f07380f8f20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f0728009fb0 tx=0x7f0728009b30).stop 2023-11-15T05:00:43.431 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.429+0000 7f0737fff700 1 -- 172.21.15.178:0/4063991205 shutdown_connections 2023-11-15T05:00:43.431 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.429+0000 7f0737fff700 1 --2- 172.21.15.178:0/4063991205 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07380601e0 0x7f07380f8f20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:43.431 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.429+0000 7f0737fff700 1 -- 172.21.15.178:0/4063991205 >> 172.21.15.178:0/4063991205 conn(0x7f07380f09a0 msgr2=0x7f07380f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:43.431 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.429+0000 7f0737fff700 1 -- 172.21.15.178:0/4063991205 shutdown_connections 2023-11-15T05:00:43.432 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.429+0000 7f0737fff700 1 -- 172.21.15.178:0/4063991205 wait complete. 2023-11-15T05:00:43.432 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.429+0000 7f0737fff700 1 Processor -- start 2023-11-15T05:00:43.432 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.429+0000 7f0737fff700 1 -- start start 2023-11-15T05:00:43.432 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.430+0000 7f0737fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07380601e0 0x7f0738106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:43.432 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.430+0000 7f0737fff700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07381069f0 con 0x7f07380601e0 2023-11-15T05:00:43.433 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.430+0000 7f0736ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07380601e0 0x7f0738106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:43.433 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.430+0000 7f0736ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07380601e0 0x7f0738106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54318/0 (socket says 172.21.15.178:54318) 2023-11-15T05:00:43.433 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.430+0000 7f0736ffd700 1 -- 172.21.15.178:0/1684792114 learned_addr learned my addr 172.21.15.178:0/1684792114 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:00:43.433 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.430+0000 7f0736ffd700 1 -- 172.21.15.178:0/1684792114 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0728005040 con 0x7f07380601e0 2023-11-15T05:00:43.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.430+0000 7f0736ffd700 1 --2- 172.21.15.178:0/1684792114 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07380601e0 0x7f0738106500 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f0728014000 tx=0x7f072800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:43.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.431+0000 7f071f7fe700 1 -- 172.21.15.178:0/1684792114 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07280095f0 con 0x7f07380601e0 2023-11-15T05:00:43.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.431+0000 7f0737fff700 1 -- 172.21.15.178:0/1684792114 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0738106bf0 con 0x7f07380601e0 2023-11-15T05:00:43.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.431+0000 7f071f7fe700 1 -- 172.21.15.178:0/1684792114 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0728007e70 con 0x7f07380601e0 2023-11-15T05:00:43.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.431+0000 7f071f7fe700 1 -- 172.21.15.178:0/1684792114 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0728014ea0 con 0x7f07380601e0 2023-11-15T05:00:43.436 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.431+0000 7f0737fff700 1 -- 172.21.15.178:0/1684792114 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0738107080 con 0x7f07380601e0 2023-11-15T05:00:43.436 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.431+0000 7f071f7fe700 1 -- 172.21.15.178:0/1684792114 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0728025070 con 0x7f07380601e0 2023-11-15T05:00:43.436 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.431+0000 7f0737fff700 1 -- 172.21.15.178:0/1684792114 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0738045ad0 con 0x7f07380601e0 2023-11-15T05:00:43.436 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.432+0000 7f071f7fe700 1 --2- 172.21.15.178:0/1684792114 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f072002ff60 0x7f0720032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:43.436 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.432+0000 7f071f7fe700 1 -- 172.21.15.178:0/1684792114 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0728040a90 con 0x7f07380601e0 2023-11-15T05:00:43.437 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.432+0000 7f07367fc700 1 --2- 172.21.15.178:0/1684792114 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f072002ff60 0x7f0720032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:43.437 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.432+0000 7f07367fc700 1 --2- 172.21.15.178:0/1684792114 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f072002ff60 0x7f0720032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f072c0096a0 tx=0x7f072c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:43.437 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.436+0000 7f071f7fe700 1 -- 172.21.15.178:0/1684792114 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0728026350 con 0x7f07380601e0 2023-11-15T05:00:43.642 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.642+0000 7f0737fff700 1 -- 172.21.15.178:0/1684792114 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0738107ed0 con 0x7f07380601e0 2023-11-15T05:00:43.643 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.642+0000 7f071f7fe700 1 -- 172.21.15.178:0/1684792114 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0728012330 con 0x7f07380601e0 2023-11-15T05:00:43.643 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:00:43.643 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:00:43.644 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.644+0000 7f0737fff700 1 -- 172.21.15.178:0/1684792114 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f072002ff60 msgr2=0x7f0720032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:43.645 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.644+0000 7f0737fff700 1 --2- 172.21.15.178:0/1684792114 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f072002ff60 0x7f0720032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f072c0096a0 tx=0x7f072c006b40).stop 2023-11-15T05:00:43.645 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.644+0000 7f0737fff700 1 -- 172.21.15.178:0/1684792114 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07380601e0 msgr2=0x7f0738106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:43.645 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.644+0000 7f0737fff700 1 --2- 172.21.15.178:0/1684792114 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07380601e0 0x7f0738106500 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f0728014000 tx=0x7f072800b040).stop 2023-11-15T05:00:43.645 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.644+0000 7f0737fff700 1 -- 172.21.15.178:0/1684792114 shutdown_connections 2023-11-15T05:00:43.645 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.644+0000 7f0737fff700 1 --2- 172.21.15.178:0/1684792114 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f072002ff60 0x7f0720032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:43.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.644+0000 7f0737fff700 1 --2- 172.21.15.178:0/1684792114 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07380601e0 0x7f0738106500 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:43.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.644+0000 7f0737fff700 1 -- 172.21.15.178:0/1684792114 >> 172.21.15.178:0/1684792114 conn(0x7f07380f09a0 msgr2=0x7f07380f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:43.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.644+0000 7f0737fff700 1 -- 172.21.15.178:0/1684792114 shutdown_connections 2023-11-15T05:00:43.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:43.644+0000 7f0737fff700 1 -- 172.21.15.178:0/1684792114 wait complete. 2023-11-15T05:00:43.647 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:00:44.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:43 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:42.358799+0000 mgr.smithi163.qesyrx (mgr.14162) 160 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:44.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:43 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:43.643076+0000 mon.smithi163 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.178:0/1684792114' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:45.023 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:45.024 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:00:46.182 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.181+0000 7fa8bbefe700 1 -- 172.21.15.178:0/3824511156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8b40f5410 msgr2=0x7fa8b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:46.182 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.181+0000 7fa8bbefe700 1 --2- 172.21.15.178:0/3824511156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8b40f5410 0x7fa8b40f5830 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fa8a4009fb0 tx=0x7fa8a4009b30).stop 2023-11-15T05:00:46.183 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.181+0000 7fa8bbefe700 1 -- 172.21.15.178:0/3824511156 shutdown_connections 2023-11-15T05:00:46.183 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.181+0000 7fa8bbefe700 1 --2- 172.21.15.178:0/3824511156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8b40f5410 0x7fa8b40f5830 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:46.183 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.181+0000 7fa8bbefe700 1 -- 172.21.15.178:0/3824511156 >> 172.21.15.178:0/3824511156 conn(0x7fa8b40f0b70 msgr2=0x7fa8b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:46.183 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.181+0000 7fa8bbefe700 1 -- 172.21.15.178:0/3824511156 shutdown_connections 2023-11-15T05:00:46.183 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.181+0000 7fa8bbefe700 1 -- 172.21.15.178:0/3824511156 wait complete. 2023-11-15T05:00:46.183 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.181+0000 7fa8bbefe700 1 Processor -- start 2023-11-15T05:00:46.184 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.181+0000 7fa8bbefe700 1 -- start start 2023-11-15T05:00:46.184 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.182+0000 7fa8bbefe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8b40f5410 0x7fa8b4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:46.184 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.182+0000 7fa8bbefe700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b40601d0 con 0x7fa8b40f5410 2023-11-15T05:00:46.184 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.182+0000 7fa8b9c9a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8b40f5410 0x7fa8b4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:46.184 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.182+0000 7fa8b9c9a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8b40f5410 0x7fa8b4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:34772/0 (socket says 172.21.15.178:34772) 2023-11-15T05:00:46.185 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.182+0000 7fa8b9c9a700 1 -- 172.21.15.178:0/2053467762 learned_addr learned my addr 172.21.15.178:0/2053467762 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:00:46.186 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.182+0000 7fa8b9c9a700 1 -- 172.21.15.178:0/2053467762 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a4005040 con 0x7fa8b40f5410 2023-11-15T05:00:46.186 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.182+0000 7fa8b9c9a700 1 --2- 172.21.15.178:0/2053467762 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8b40f5410 0x7fa8b4109430 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fa8a4013510 tx=0x7fa8a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:46.186 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.182+0000 7fa8aa7fc700 1 -- 172.21.15.178:0/2053467762 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8a4009320 con 0x7fa8b40f5410 2023-11-15T05:00:46.187 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.182+0000 7fa8bbefe700 1 -- 172.21.15.178:0/2053467762 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b40603d0 con 0x7fa8b40f5410 2023-11-15T05:00:46.187 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.183+0000 7fa8aa7fc700 1 -- 172.21.15.178:0/2053467762 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8a4013f00 con 0x7fa8b40f5410 2023-11-15T05:00:46.187 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.183+0000 7fa8aa7fc700 1 -- 172.21.15.178:0/2053467762 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8a4014d90 con 0x7fa8b40f5410 2023-11-15T05:00:46.187 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.183+0000 7fa8bbefe700 1 -- 172.21.15.178:0/2053467762 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b4109ae0 con 0x7fa8b40f5410 2023-11-15T05:00:46.187 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.183+0000 7fa8aa7fc700 1 -- 172.21.15.178:0/2053467762 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8a40095f0 con 0x7fa8b40f5410 2023-11-15T05:00:46.188 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.183+0000 7fa8bbefe700 1 -- 172.21.15.178:0/2053467762 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b4045ad0 con 0x7fa8b40f5410 2023-11-15T05:00:46.188 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.183+0000 7fa8aa7fc700 1 --2- 172.21.15.178:0/2053467762 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa8a002ff60 0x7fa8a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:46.188 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.183+0000 7fa8aa7fc700 1 -- 172.21.15.178:0/2053467762 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8a40410c0 con 0x7fa8b40f5410 2023-11-15T05:00:46.188 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.184+0000 7fa8b9499700 1 --2- 172.21.15.178:0/2053467762 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa8a002ff60 0x7fa8a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:46.188 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.184+0000 7fa8b9499700 1 --2- 172.21.15.178:0/2053467762 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa8a002ff60 0x7fa8a0032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fa8b00096a0 tx=0x7fa8b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:46.189 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.188+0000 7fa8aa7fc700 1 -- 172.21.15.178:0/2053467762 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8a4014420 con 0x7fa8b40f5410 2023-11-15T05:00:46.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:45 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:44.359067+0000 mgr.smithi163.qesyrx (mgr.14162) 161 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:46.393 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.392+0000 7fa8bbefe700 1 -- 172.21.15.178:0/2053467762 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8b410a2a0 con 0x7fa8b40f5410 2023-11-15T05:00:46.394 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.393+0000 7fa8aa7fc700 1 -- 172.21.15.178:0/2053467762 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa8a4010230 con 0x7fa8b40f5410 2023-11-15T05:00:46.394 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:00:46.395 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:00:46.396 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.395+0000 7fa8bbefe700 1 -- 172.21.15.178:0/2053467762 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa8a002ff60 msgr2=0x7fa8a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:46.396 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.395+0000 7fa8bbefe700 1 --2- 172.21.15.178:0/2053467762 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa8a002ff60 0x7fa8a0032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fa8b00096a0 tx=0x7fa8b0006b40).stop 2023-11-15T05:00:46.397 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.395+0000 7fa8bbefe700 1 -- 172.21.15.178:0/2053467762 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8b40f5410 msgr2=0x7fa8b4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:46.397 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.395+0000 7fa8bbefe700 1 --2- 172.21.15.178:0/2053467762 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8b40f5410 0x7fa8b4109430 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fa8a4013510 tx=0x7fa8a400b040).stop 2023-11-15T05:00:46.398 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.395+0000 7fa8bbefe700 1 -- 172.21.15.178:0/2053467762 shutdown_connections 2023-11-15T05:00:46.398 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.395+0000 7fa8bbefe700 1 --2- 172.21.15.178:0/2053467762 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa8a002ff60 0x7fa8a0032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:46.399 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.395+0000 7fa8bbefe700 1 --2- 172.21.15.178:0/2053467762 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8b40f5410 0x7fa8b4109430 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:46.399 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.395+0000 7fa8bbefe700 1 -- 172.21.15.178:0/2053467762 >> 172.21.15.178:0/2053467762 conn(0x7fa8b40f0b70 msgr2=0x7fa8b40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:46.399 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.395+0000 7fa8bbefe700 1 -- 172.21.15.178:0/2053467762 shutdown_connections 2023-11-15T05:00:46.400 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:46.395+0000 7fa8bbefe700 1 -- 172.21.15.178:0/2053467762 wait complete. 2023-11-15T05:00:46.400 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:00:47.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:46 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:46.393684+0000 mon.smithi163 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.178:0/2053467762' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:47.786 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:47.787 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:00:48.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:47 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:46.359327+0000 mgr.smithi163.qesyrx (mgr.14162) 162 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:48.978 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.978+0000 7f1d87f58700 1 -- 172.21.15.178:0/1239109361 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d800f4880 msgr2=0x7f1d800f4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:48.978 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.978+0000 7f1d87f58700 1 --2- 172.21.15.178:0/1239109361 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d800f4880 0x7f1d800f4ca0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f1d74009fb0 tx=0x7f1d74009b30).stop 2023-11-15T05:00:48.979 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.978+0000 7f1d87f58700 1 -- 172.21.15.178:0/1239109361 shutdown_connections 2023-11-15T05:00:48.979 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.978+0000 7f1d87f58700 1 --2- 172.21.15.178:0/1239109361 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d800f4880 0x7f1d800f4ca0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:48.979 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.978+0000 7f1d87f58700 1 -- 172.21.15.178:0/1239109361 >> 172.21.15.178:0/1239109361 conn(0x7f1d800f02e0 msgr2=0x7f1d800f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:48.979 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.978+0000 7f1d87f58700 1 -- 172.21.15.178:0/1239109361 shutdown_connections 2023-11-15T05:00:48.979 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.978+0000 7f1d87f58700 1 -- 172.21.15.178:0/1239109361 wait complete. 2023-11-15T05:00:48.981 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.978+0000 7f1d87f58700 1 Processor -- start 2023-11-15T05:00:48.981 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.978+0000 7f1d87f58700 1 -- start start 2023-11-15T05:00:48.982 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.979+0000 7f1d87f58700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d800f4880 0x7f1d801066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:48.982 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.979+0000 7f1d87f58700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d80106ba0 con 0x7f1d800f4880 2023-11-15T05:00:48.982 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.979+0000 7f1d85cf4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d800f4880 0x7f1d801066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:48.982 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.979+0000 7f1d85cf4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d800f4880 0x7f1d801066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:34796/0 (socket says 172.21.15.178:34796) 2023-11-15T05:00:48.982 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.979+0000 7f1d85cf4700 1 -- 172.21.15.178:0/1401849359 learned_addr learned my addr 172.21.15.178:0/1401849359 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:00:48.983 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.979+0000 7f1d85cf4700 1 -- 172.21.15.178:0/1401849359 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d74005040 con 0x7f1d800f4880 2023-11-15T05:00:48.983 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.979+0000 7f1d85cf4700 1 --2- 172.21.15.178:0/1401849359 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d800f4880 0x7f1d801066b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f1d74013510 tx=0x7f1d7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:48.983 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.979+0000 7f1d727fc700 1 -- 172.21.15.178:0/1401849359 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d74009320 con 0x7f1d800f4880 2023-11-15T05:00:48.983 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.980+0000 7f1d87f58700 1 -- 172.21.15.178:0/1401849359 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d80106da0 con 0x7f1d800f4880 2023-11-15T05:00:48.983 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.980+0000 7f1d727fc700 1 -- 172.21.15.178:0/1401849359 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d74013f00 con 0x7f1d800f4880 2023-11-15T05:00:48.984 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.980+0000 7f1d727fc700 1 -- 172.21.15.178:0/1401849359 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d74014dc0 con 0x7f1d800f4880 2023-11-15T05:00:48.984 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.980+0000 7f1d87f58700 1 -- 172.21.15.178:0/1401849359 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d80107230 con 0x7f1d800f4880 2023-11-15T05:00:48.984 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.980+0000 7f1d727fc700 1 -- 172.21.15.178:0/1401849359 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1d7401d3d0 con 0x7f1d800f4880 2023-11-15T05:00:48.984 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.980+0000 7f1d727fc700 1 --2- 172.21.15.178:0/1401849359 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1d6c02ff60 0x7f1d6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:48.984 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.980+0000 7f1d727fc700 1 -- 172.21.15.178:0/1401849359 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d74041080 con 0x7f1d800f4880 2023-11-15T05:00:48.985 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.980+0000 7f1d87f58700 1 -- 172.21.15.178:0/1401849359 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d80045ad0 con 0x7f1d800f4880 2023-11-15T05:00:48.985 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.981+0000 7f1d854f3700 1 --2- 172.21.15.178:0/1401849359 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1d6c02ff60 0x7f1d6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:48.985 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.981+0000 7f1d854f3700 1 --2- 172.21.15.178:0/1401849359 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1d6c02ff60 0x7f1d6c032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f1d7c0096a0 tx=0x7f1d7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:48.987 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:48.985+0000 7f1d727fc700 1 -- 172.21.15.178:0/1401849359 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d74012900 con 0x7f1d800f4880 2023-11-15T05:00:49.191 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:49.191+0000 7f1d87f58700 1 -- 172.21.15.178:0/1401849359 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d800fc040 con 0x7f1d800f4880 2023-11-15T05:00:49.192 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:49.191+0000 7f1d727fc700 1 -- 172.21.15.178:0/1401849359 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1d7402b020 con 0x7f1d800f4880 2023-11-15T05:00:49.192 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:00:49.192 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:00:49.193 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:49.193+0000 7f1d87f58700 1 -- 172.21.15.178:0/1401849359 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1d6c02ff60 msgr2=0x7f1d6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:49.193 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:49.193+0000 7f1d87f58700 1 --2- 172.21.15.178:0/1401849359 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1d6c02ff60 0x7f1d6c032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f1d7c0096a0 tx=0x7f1d7c006b40).stop 2023-11-15T05:00:49.194 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:49.193+0000 7f1d87f58700 1 -- 172.21.15.178:0/1401849359 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d800f4880 msgr2=0x7f1d801066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:49.194 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:49.193+0000 7f1d87f58700 1 --2- 172.21.15.178:0/1401849359 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d800f4880 0x7f1d801066b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f1d74013510 tx=0x7f1d7400b040).stop 2023-11-15T05:00:49.194 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:49.193+0000 7f1d87f58700 1 -- 172.21.15.178:0/1401849359 shutdown_connections 2023-11-15T05:00:49.195 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:49.193+0000 7f1d87f58700 1 --2- 172.21.15.178:0/1401849359 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1d6c02ff60 0x7f1d6c032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:49.195 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:49.193+0000 7f1d87f58700 1 --2- 172.21.15.178:0/1401849359 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d800f4880 0x7f1d801066b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:49.195 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:49.193+0000 7f1d87f58700 1 -- 172.21.15.178:0/1401849359 >> 172.21.15.178:0/1401849359 conn(0x7f1d800f02e0 msgr2=0x7f1d800f0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:49.195 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:49.193+0000 7f1d87f58700 1 -- 172.21.15.178:0/1401849359 shutdown_connections 2023-11-15T05:00:49.196 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:49.193+0000 7f1d87f58700 1 -- 172.21.15.178:0/1401849359 wait complete. 2023-11-15T05:00:49.196 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:00:50.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:49 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:48.359662+0000 mgr.smithi163.qesyrx (mgr.14162) 163 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:50.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:49 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:49.191922+0000 mon.smithi163 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.178:0/1401849359' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:50.583 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:50.583 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:00:51.715 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.714+0000 7f5c4081d700 1 -- 172.21.15.178:0/2827239871 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c3c0f51a0 msgr2=0x7f5c3c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:51.716 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.714+0000 7f5c4081d700 1 --2- 172.21.15.178:0/2827239871 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c3c0f51a0 0x7f5c3c0f55c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f5c30009fb0 tx=0x7f5c30009b30).stop 2023-11-15T05:00:51.716 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.714+0000 7f5c4081d700 1 -- 172.21.15.178:0/2827239871 shutdown_connections 2023-11-15T05:00:51.716 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.714+0000 7f5c4081d700 1 --2- 172.21.15.178:0/2827239871 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c3c0f51a0 0x7f5c3c0f55c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:51.716 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.714+0000 7f5c4081d700 1 -- 172.21.15.178:0/2827239871 >> 172.21.15.178:0/2827239871 conn(0x7f5c3c0f09a0 msgr2=0x7f5c3c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:51.717 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.714+0000 7f5c4081d700 1 -- 172.21.15.178:0/2827239871 shutdown_connections 2023-11-15T05:00:51.717 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.714+0000 7f5c4081d700 1 -- 172.21.15.178:0/2827239871 wait complete. 2023-11-15T05:00:51.717 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.715+0000 7f5c4081d700 1 Processor -- start 2023-11-15T05:00:51.717 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.715+0000 7f5c4081d700 1 -- start start 2023-11-15T05:00:51.717 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.715+0000 7f5c4081d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c3c0f51a0 0x7f5c3c1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:51.717 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.715+0000 7f5c4081d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c060220 con 0x7f5c3c0f51a0 2023-11-15T05:00:51.718 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.715+0000 7f5c3ad9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c3c0f51a0 0x7f5c3c1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:51.718 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.715+0000 7f5c3ad9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c3c0f51a0 0x7f5c3c1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:34820/0 (socket says 172.21.15.178:34820) 2023-11-15T05:00:51.718 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.715+0000 7f5c3ad9d700 1 -- 172.21.15.178:0/1250555564 learned_addr learned my addr 172.21.15.178:0/1250555564 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:00:51.719 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.716+0000 7f5c3ad9d700 1 -- 172.21.15.178:0/1250555564 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c30005040 con 0x7f5c3c0f51a0 2023-11-15T05:00:51.720 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.716+0000 7f5c3ad9d700 1 --2- 172.21.15.178:0/1250555564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c3c0f51a0 0x7f5c3c1091d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f5c30013510 tx=0x7f5c3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:51.720 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.716+0000 7f5c237fe700 1 -- 172.21.15.178:0/1250555564 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c30009320 con 0x7f5c3c0f51a0 2023-11-15T05:00:51.720 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.716+0000 7f5c4081d700 1 -- 172.21.15.178:0/1250555564 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c3c060420 con 0x7f5c3c0f51a0 2023-11-15T05:00:51.720 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.716+0000 7f5c237fe700 1 -- 172.21.15.178:0/1250555564 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c30013f00 con 0x7f5c3c0f51a0 2023-11-15T05:00:51.720 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.716+0000 7f5c237fe700 1 -- 172.21.15.178:0/1250555564 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c30014dc0 con 0x7f5c3c0f51a0 2023-11-15T05:00:51.721 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.716+0000 7f5c4081d700 1 -- 172.21.15.178:0/1250555564 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c3c109880 con 0x7f5c3c0f51a0 2023-11-15T05:00:51.721 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.717+0000 7f5c237fe700 1 -- 172.21.15.178:0/1250555564 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5c3001d3d0 con 0x7f5c3c0f51a0 2023-11-15T05:00:51.721 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.717+0000 7f5c4081d700 1 -- 172.21.15.178:0/1250555564 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c3c059100 con 0x7f5c3c0f51a0 2023-11-15T05:00:51.721 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.717+0000 7f5c237fe700 1 --2- 172.21.15.178:0/1250555564 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5c2402ff60 0x7f5c24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:51.721 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.717+0000 7f5c237fe700 1 -- 172.21.15.178:0/1250555564 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5c30041320 con 0x7f5c3c0f51a0 2023-11-15T05:00:51.722 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.717+0000 7f5c3a59c700 1 --2- 172.21.15.178:0/1250555564 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5c2402ff60 0x7f5c24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:51.722 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.717+0000 7f5c3a59c700 1 --2- 172.21.15.178:0/1250555564 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5c2402ff60 0x7f5c24032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f5c2c0096a0 tx=0x7f5c2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:51.722 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.722+0000 7f5c237fe700 1 -- 172.21.15.178:0/1250555564 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c3001cb10 con 0x7f5c3c0f51a0 2023-11-15T05:00:51.927 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.926+0000 7f5c4081d700 1 -- 172.21.15.178:0/1250555564 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c3c045ad0 con 0x7f5c3c0f51a0 2023-11-15T05:00:51.927 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.927+0000 7f5c237fe700 1 -- 172.21.15.178:0/1250555564 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5c3001cb10 con 0x7f5c3c0f51a0 2023-11-15T05:00:51.928 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:00:51.928 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:00:51.929 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.928+0000 7f5c4081d700 1 -- 172.21.15.178:0/1250555564 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5c2402ff60 msgr2=0x7f5c24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:51.929 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.928+0000 7f5c4081d700 1 --2- 172.21.15.178:0/1250555564 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5c2402ff60 0x7f5c24032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f5c2c0096a0 tx=0x7f5c2c006b40).stop 2023-11-15T05:00:51.929 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.929+0000 7f5c4081d700 1 -- 172.21.15.178:0/1250555564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c3c0f51a0 msgr2=0x7f5c3c1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:51.930 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.929+0000 7f5c4081d700 1 --2- 172.21.15.178:0/1250555564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c3c0f51a0 0x7f5c3c1091d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f5c30013510 tx=0x7f5c3000b040).stop 2023-11-15T05:00:51.930 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.929+0000 7f5c4081d700 1 -- 172.21.15.178:0/1250555564 shutdown_connections 2023-11-15T05:00:51.930 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.929+0000 7f5c4081d700 1 --2- 172.21.15.178:0/1250555564 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5c2402ff60 0x7f5c24032420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:51.930 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.929+0000 7f5c4081d700 1 --2- 172.21.15.178:0/1250555564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c3c0f51a0 0x7f5c3c1091d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:51.930 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.929+0000 7f5c4081d700 1 -- 172.21.15.178:0/1250555564 >> 172.21.15.178:0/1250555564 conn(0x7f5c3c0f09a0 msgr2=0x7f5c3c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:51.931 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.929+0000 7f5c4081d700 1 -- 172.21.15.178:0/1250555564 shutdown_connections 2023-11-15T05:00:51.931 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:51.929+0000 7f5c4081d700 1 -- 172.21.15.178:0/1250555564 wait complete. 2023-11-15T05:00:51.931 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:00:52.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:51 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:50.359944+0000 mgr.smithi163.qesyrx (mgr.14162) 164 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:53.205 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:52 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:51.927738+0000 mon.smithi163 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.178:0/1250555564' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:53.354 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:53.355 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:00:54.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:53 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:52.360288+0000 mgr.smithi163.qesyrx (mgr.14162) 165 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:54.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.490+0000 7ff0ff618700 1 -- 172.21.15.178:0/3241745430 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0f80f4b80 msgr2=0x7ff0f80f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:54.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.490+0000 7ff0ff618700 1 --2- 172.21.15.178:0/3241745430 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0f80f4b80 0x7ff0f80f4fa0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7ff0e8009fb0 tx=0x7ff0e8009b30).stop 2023-11-15T05:00:54.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.490+0000 7ff0ff618700 1 -- 172.21.15.178:0/3241745430 shutdown_connections 2023-11-15T05:00:54.492 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.490+0000 7ff0ff618700 1 --2- 172.21.15.178:0/3241745430 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0f80f4b80 0x7ff0f80f4fa0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:54.492 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.491+0000 7ff0ff618700 1 -- 172.21.15.178:0/3241745430 >> 172.21.15.178:0/3241745430 conn(0x7ff0f80f02e0 msgr2=0x7ff0f80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:54.492 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.491+0000 7ff0ff618700 1 -- 172.21.15.178:0/3241745430 shutdown_connections 2023-11-15T05:00:54.492 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.491+0000 7ff0ff618700 1 -- 172.21.15.178:0/3241745430 wait complete. 2023-11-15T05:00:54.494 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.491+0000 7ff0ff618700 1 Processor -- start 2023-11-15T05:00:54.495 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.491+0000 7ff0ff618700 1 -- start start 2023-11-15T05:00:54.495 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.492+0000 7ff0ff618700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0f80f4b80 0x7ff0f8106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:54.495 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.492+0000 7ff0ff618700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f8106c10 con 0x7ff0f80f4b80 2023-11-15T05:00:54.495 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.492+0000 7ff0fd3b4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0f80f4b80 0x7ff0f8106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:54.495 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.492+0000 7ff0fd3b4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0f80f4b80 0x7ff0f8106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:38528/0 (socket says 172.21.15.178:38528) 2023-11-15T05:00:54.496 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.492+0000 7ff0fd3b4700 1 -- 172.21.15.178:0/1713193720 learned_addr learned my addr 172.21.15.178:0/1713193720 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:00:54.496 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.492+0000 7ff0fd3b4700 1 -- 172.21.15.178:0/1713193720 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e8005040 con 0x7ff0f80f4b80 2023-11-15T05:00:54.496 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.492+0000 7ff0fd3b4700 1 --2- 172.21.15.178:0/1713193720 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0f80f4b80 0x7ff0f8106720 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7ff0e8013510 tx=0x7ff0e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:54.496 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.492+0000 7ff0edffb700 1 -- 172.21.15.178:0/1713193720 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0e8009320 con 0x7ff0f80f4b80 2023-11-15T05:00:54.496 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.492+0000 7ff0ff618700 1 -- 172.21.15.178:0/1713193720 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f8106e10 con 0x7ff0f80f4b80 2023-11-15T05:00:54.497 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.492+0000 7ff0edffb700 1 -- 172.21.15.178:0/1713193720 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff0e8013f00 con 0x7ff0f80f4b80 2023-11-15T05:00:54.497 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.492+0000 7ff0edffb700 1 -- 172.21.15.178:0/1713193720 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0e8014d90 con 0x7ff0f80f4b80 2023-11-15T05:00:54.497 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.493+0000 7ff0ff618700 1 -- 172.21.15.178:0/1713193720 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f81072a0 con 0x7ff0f80f4b80 2023-11-15T05:00:54.497 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.493+0000 7ff0edffb700 1 -- 172.21.15.178:0/1713193720 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff0e8014420 con 0x7ff0f80f4b80 2023-11-15T05:00:54.497 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.493+0000 7ff0ff618700 1 -- 172.21.15.178:0/1713193720 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f8045ad0 con 0x7ff0f80f4b80 2023-11-15T05:00:54.498 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.493+0000 7ff0edffb700 1 --2- 172.21.15.178:0/1713193720 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff0e402ff50 0x7ff0e4032410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:54.498 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.493+0000 7ff0edffb700 1 -- 172.21.15.178:0/1713193720 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff0e8041050 con 0x7ff0f80f4b80 2023-11-15T05:00:54.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.494+0000 7ff0fcbb3700 1 --2- 172.21.15.178:0/1713193720 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff0e402ff50 0x7ff0e4032410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:54.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.494+0000 7ff0fcbb3700 1 --2- 172.21.15.178:0/1713193720 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff0e402ff50 0x7ff0e4032410 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7ff0f40096a0 tx=0x7ff0f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:54.500 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.498+0000 7ff0edffb700 1 -- 172.21.15.178:0/1713193720 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff0e8012330 con 0x7ff0f80f4b80 2023-11-15T05:00:54.704 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.704+0000 7ff0ff618700 1 -- 172.21.15.178:0/1713193720 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff0f80f9d40 con 0x7ff0f80f4b80 2023-11-15T05:00:54.705 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.705+0000 7ff0edffb700 1 -- 172.21.15.178:0/1713193720 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff0e801db70 con 0x7ff0f80f4b80 2023-11-15T05:00:54.705 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:00:54.706 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:00:54.707 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.706+0000 7ff0ff618700 1 -- 172.21.15.178:0/1713193720 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff0e402ff50 msgr2=0x7ff0e4032410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:54.707 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.706+0000 7ff0ff618700 1 --2- 172.21.15.178:0/1713193720 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff0e402ff50 0x7ff0e4032410 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7ff0f40096a0 tx=0x7ff0f4006b40).stop 2023-11-15T05:00:54.707 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.706+0000 7ff0ff618700 1 -- 172.21.15.178:0/1713193720 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0f80f4b80 msgr2=0x7ff0f8106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:54.707 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.706+0000 7ff0ff618700 1 --2- 172.21.15.178:0/1713193720 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0f80f4b80 0x7ff0f8106720 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7ff0e8013510 tx=0x7ff0e800b040).stop 2023-11-15T05:00:54.708 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.706+0000 7ff0ff618700 1 -- 172.21.15.178:0/1713193720 shutdown_connections 2023-11-15T05:00:54.708 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.706+0000 7ff0ff618700 1 --2- 172.21.15.178:0/1713193720 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff0e402ff50 0x7ff0e4032410 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:54.708 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.706+0000 7ff0ff618700 1 --2- 172.21.15.178:0/1713193720 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0f80f4b80 0x7ff0f8106720 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:54.708 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.706+0000 7ff0ff618700 1 -- 172.21.15.178:0/1713193720 >> 172.21.15.178:0/1713193720 conn(0x7ff0f80f02e0 msgr2=0x7ff0f80f0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:54.708 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.706+0000 7ff0ff618700 1 -- 172.21.15.178:0/1713193720 shutdown_connections 2023-11-15T05:00:54.709 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:54.707+0000 7ff0ff618700 1 -- 172.21.15.178:0/1713193720 wait complete. 2023-11-15T05:00:54.709 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:00:55.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:54 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:54.705529+0000 mon.smithi163 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.178:0/1713193720' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:56.113 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:56.114 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:00:56.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:55 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:54.360575+0000 mgr.smithi163.qesyrx (mgr.14162) 166 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:57.337 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.337+0000 7fa91b1a1700 1 -- 172.21.15.178:0/30079432 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9140601c0 msgr2=0x7fa9140f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:57.338 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.337+0000 7fa91b1a1700 1 --2- 172.21.15.178:0/30079432 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9140601c0 0x7fa9140f91b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fa908009fb0 tx=0x7fa908009b30).stop 2023-11-15T05:00:57.338 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.337+0000 7fa91b1a1700 1 -- 172.21.15.178:0/30079432 shutdown_connections 2023-11-15T05:00:57.338 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.337+0000 7fa91b1a1700 1 --2- 172.21.15.178:0/30079432 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9140601c0 0x7fa9140f91b0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:57.338 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.337+0000 7fa91b1a1700 1 -- 172.21.15.178:0/30079432 >> 172.21.15.178:0/30079432 conn(0x7fa9140f0b70 msgr2=0x7fa9140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:57.339 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.337+0000 7fa91b1a1700 1 -- 172.21.15.178:0/30079432 shutdown_connections 2023-11-15T05:00:57.339 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.337+0000 7fa91b1a1700 1 -- 172.21.15.178:0/30079432 wait complete. 2023-11-15T05:00:57.339 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.337+0000 7fa91b1a1700 1 Processor -- start 2023-11-15T05:00:57.339 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.337+0000 7fa91b1a1700 1 -- start start 2023-11-15T05:00:57.339 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.337+0000 7fa91b1a1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9140601c0 0x7fa9140f6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:57.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.337+0000 7fa91b1a1700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9140f50b0 con 0x7fa9140601c0 2023-11-15T05:00:57.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.338+0000 7fa918f3d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9140601c0 0x7fa9140f6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:57.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.338+0000 7fa918f3d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9140601c0 0x7fa9140f6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:38538/0 (socket says 172.21.15.178:38538) 2023-11-15T05:00:57.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.338+0000 7fa918f3d700 1 -- 172.21.15.178:0/182395245 learned_addr learned my addr 172.21.15.178:0/182395245 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:00:57.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.338+0000 7fa918f3d700 1 -- 172.21.15.178:0/182395245 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa908005040 con 0x7fa9140601c0 2023-11-15T05:00:57.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.338+0000 7fa918f3d700 1 --2- 172.21.15.178:0/182395245 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9140601c0 0x7fa9140f6c30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fa908013510 tx=0x7fa90800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:57.343 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.338+0000 7fa9117fa700 1 -- 172.21.15.178:0/182395245 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa908009320 con 0x7fa9140601c0 2023-11-15T05:00:57.343 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.338+0000 7fa91b1a1700 1 -- 172.21.15.178:0/182395245 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9140f5250 con 0x7fa9140601c0 2023-11-15T05:00:57.343 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.338+0000 7fa9117fa700 1 -- 172.21.15.178:0/182395245 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa908013f00 con 0x7fa9140601c0 2023-11-15T05:00:57.343 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.338+0000 7fa9117fa700 1 -- 172.21.15.178:0/182395245 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa908014dc0 con 0x7fa9140601c0 2023-11-15T05:00:57.343 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.338+0000 7fa91b1a1700 1 -- 172.21.15.178:0/182395245 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9140f56e0 con 0x7fa9140601c0 2023-11-15T05:00:57.344 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.339+0000 7fa9117fa700 1 -- 172.21.15.178:0/182395245 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9080095f0 con 0x7fa9140601c0 2023-11-15T05:00:57.344 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.339+0000 7fa91b1a1700 1 -- 172.21.15.178:0/182395245 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa914059100 con 0x7fa9140601c0 2023-11-15T05:00:57.344 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.339+0000 7fa9117fa700 1 --2- 172.21.15.178:0/182395245 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa90402ff60 0x7fa904032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:57.344 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.339+0000 7fa9117fa700 1 -- 172.21.15.178:0/182395245 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa908041020 con 0x7fa9140601c0 2023-11-15T05:00:57.345 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.339+0000 7fa913fff700 1 --2- 172.21.15.178:0/182395245 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa90402ff60 0x7fa904032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:57.345 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.340+0000 7fa913fff700 1 --2- 172.21.15.178:0/182395245 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa90402ff60 0x7fa904032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fa9000096a0 tx=0x7fa900006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:57.345 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.344+0000 7fa9117fa700 1 -- 172.21.15.178:0/182395245 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa90806d2b0 con 0x7fa9140601c0 2023-11-15T05:00:57.549 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.549+0000 7fa91b1a1700 1 -- 172.21.15.178:0/182395245 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa914045ad0 con 0x7fa9140601c0 2023-11-15T05:00:57.550 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.549+0000 7fa9117fa700 1 -- 172.21.15.178:0/182395245 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa90801c920 con 0x7fa9140601c0 2023-11-15T05:00:57.550 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:00:57.550 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:00:57.551 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.551+0000 7fa91b1a1700 1 -- 172.21.15.178:0/182395245 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa90402ff60 msgr2=0x7fa904032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:57.551 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.551+0000 7fa91b1a1700 1 --2- 172.21.15.178:0/182395245 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa90402ff60 0x7fa904032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fa9000096a0 tx=0x7fa900006b40).stop 2023-11-15T05:00:57.552 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.551+0000 7fa91b1a1700 1 -- 172.21.15.178:0/182395245 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9140601c0 msgr2=0x7fa9140f6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:57.552 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.551+0000 7fa91b1a1700 1 --2- 172.21.15.178:0/182395245 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9140601c0 0x7fa9140f6c30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fa908013510 tx=0x7fa90800b040).stop 2023-11-15T05:00:57.552 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.551+0000 7fa91b1a1700 1 -- 172.21.15.178:0/182395245 shutdown_connections 2023-11-15T05:00:57.552 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.551+0000 7fa91b1a1700 1 --2- 172.21.15.178:0/182395245 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa90402ff60 0x7fa904032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:57.552 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.551+0000 7fa91b1a1700 1 --2- 172.21.15.178:0/182395245 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9140601c0 0x7fa9140f6c30 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:57.553 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.551+0000 7fa91b1a1700 1 -- 172.21.15.178:0/182395245 >> 172.21.15.178:0/182395245 conn(0x7fa9140f0b70 msgr2=0x7fa9140f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:57.553 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.551+0000 7fa91b1a1700 1 -- 172.21.15.178:0/182395245 shutdown_connections 2023-11-15T05:00:57.553 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:00:57.551+0000 7fa91b1a1700 1 -- 172.21.15.178:0/182395245 wait complete. 2023-11-15T05:00:57.554 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:00:58.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:57 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:56.360850+0000 mgr.smithi163.qesyrx (mgr.14162) 167 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:00:58.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:57 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:00:57.550048+0000 mon.smithi163 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.178:0/182395245' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:00:58.943 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:00:58.944 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:00.201 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.200+0000 7f47c2962700 1 -- 172.21.15.178:0/3473589377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47bc0f6510 msgr2=0x7f47bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:00.201 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.200+0000 7f47c2962700 1 --2- 172.21.15.178:0/3473589377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47bc0f6510 0x7f47bc0f6930 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f47ac009fb0 tx=0x7f47ac009b30).stop 2023-11-15T05:01:00.202 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.200+0000 7f47c2962700 1 -- 172.21.15.178:0/3473589377 shutdown_connections 2023-11-15T05:01:00.202 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.200+0000 7f47c2962700 1 --2- 172.21.15.178:0/3473589377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47bc0f6510 0x7f47bc0f6930 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:00.202 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.200+0000 7f47c2962700 1 -- 172.21.15.178:0/3473589377 >> 172.21.15.178:0/3473589377 conn(0x7f47bc0f1c70 msgr2=0x7f47bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:00.202 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.200+0000 7f47c2962700 1 -- 172.21.15.178:0/3473589377 shutdown_connections 2023-11-15T05:01:00.203 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.200+0000 7f47c2962700 1 -- 172.21.15.178:0/3473589377 wait complete. 2023-11-15T05:01:00.203 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.201+0000 7f47c2962700 1 Processor -- start 2023-11-15T05:01:00.203 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.201+0000 7f47c2962700 1 -- start start 2023-11-15T05:01:00.204 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:00:59 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:00:58.361197+0000 mgr.smithi163.qesyrx (mgr.14162) 168 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:00.206 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.201+0000 7f47c2962700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47bc0f6510 0x7f47bc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:00.206 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.201+0000 7f47c2962700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc106c20 con 0x7f47bc0f6510 2023-11-15T05:01:00.206 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.202+0000 7f47bbfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47bc0f6510 0x7f47bc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:00.206 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.202+0000 7f47bbfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47bc0f6510 0x7f47bc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:38558/0 (socket says 172.21.15.178:38558) 2023-11-15T05:01:00.207 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.202+0000 7f47bbfff700 1 -- 172.21.15.178:0/245916470 learned_addr learned my addr 172.21.15.178:0/245916470 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:00.207 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.202+0000 7f47bbfff700 1 -- 172.21.15.178:0/245916470 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ac005040 con 0x7f47bc0f6510 2023-11-15T05:01:00.207 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.202+0000 7f47bbfff700 1 --2- 172.21.15.178:0/245916470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47bc0f6510 0x7f47bc106730 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f47ac013510 tx=0x7f47ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:00.207 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.202+0000 7f47b8ff9700 1 -- 172.21.15.178:0/245916470 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47ac009320 con 0x7f47bc0f6510 2023-11-15T05:01:00.207 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.202+0000 7f47c2962700 1 -- 172.21.15.178:0/245916470 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47bc106e20 con 0x7f47bc0f6510 2023-11-15T05:01:00.208 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.202+0000 7f47b8ff9700 1 -- 172.21.15.178:0/245916470 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47ac013f00 con 0x7f47bc0f6510 2023-11-15T05:01:00.208 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.202+0000 7f47b8ff9700 1 -- 172.21.15.178:0/245916470 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47ac014d90 con 0x7f47bc0f6510 2023-11-15T05:01:00.208 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.202+0000 7f47c2962700 1 -- 172.21.15.178:0/245916470 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47bc1072b0 con 0x7f47bc0f6510 2023-11-15T05:01:00.208 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.203+0000 7f47b8ff9700 1 -- 172.21.15.178:0/245916470 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f47ac0263a0 con 0x7f47bc0f6510 2023-11-15T05:01:00.208 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.203+0000 7f47b8ff9700 1 --2- 172.21.15.178:0/245916470 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f47a402ff60 0x7f47a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:00.209 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.203+0000 7f47b8ff9700 1 -- 172.21.15.178:0/245916470 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f47ac0410a0 con 0x7f47bc0f6510 2023-11-15T05:01:00.209 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.203+0000 7f47c2962700 1 -- 172.21.15.178:0/245916470 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47bc045ad0 con 0x7f47bc0f6510 2023-11-15T05:01:00.209 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.203+0000 7f47bb7fe700 1 --2- 172.21.15.178:0/245916470 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f47a402ff60 0x7f47a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:00.209 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.204+0000 7f47bb7fe700 1 --2- 172.21.15.178:0/245916470 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f47a402ff60 0x7f47a4032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f47b00096a0 tx=0x7f47b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:00.210 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.208+0000 7f47b8ff9700 1 -- 172.21.15.178:0/245916470 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47ac012920 con 0x7f47bc0f6510 2023-11-15T05:01:00.411 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.411+0000 7f47c2962700 1 -- 172.21.15.178:0/245916470 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47bc0fc050 con 0x7f47bc0f6510 2023-11-15T05:01:00.412 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.412+0000 7f47b8ff9700 1 -- 172.21.15.178:0/245916470 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f47ac02b020 con 0x7f47bc0f6510 2023-11-15T05:01:00.412 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:00.412 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:00.413 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.413+0000 7f47c2962700 1 -- 172.21.15.178:0/245916470 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f47a402ff60 msgr2=0x7f47a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:00.414 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.413+0000 7f47c2962700 1 --2- 172.21.15.178:0/245916470 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f47a402ff60 0x7f47a4032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f47b00096a0 tx=0x7f47b0006b40).stop 2023-11-15T05:01:00.414 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.413+0000 7f47c2962700 1 -- 172.21.15.178:0/245916470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47bc0f6510 msgr2=0x7f47bc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:00.414 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.413+0000 7f47c2962700 1 --2- 172.21.15.178:0/245916470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47bc0f6510 0x7f47bc106730 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f47ac013510 tx=0x7f47ac00b040).stop 2023-11-15T05:01:00.414 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.413+0000 7f47c2962700 1 -- 172.21.15.178:0/245916470 shutdown_connections 2023-11-15T05:01:00.415 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.413+0000 7f47c2962700 1 --2- 172.21.15.178:0/245916470 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f47a402ff60 0x7f47a4032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:00.415 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.413+0000 7f47c2962700 1 --2- 172.21.15.178:0/245916470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47bc0f6510 0x7f47bc106730 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:00.415 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.413+0000 7f47c2962700 1 -- 172.21.15.178:0/245916470 >> 172.21.15.178:0/245916470 conn(0x7f47bc0f1c70 msgr2=0x7f47bc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:00.415 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.413+0000 7f47c2962700 1 -- 172.21.15.178:0/245916470 shutdown_connections 2023-11-15T05:01:00.415 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:00.413+0000 7f47c2962700 1 -- 172.21.15.178:0/245916470 wait complete. 2023-11-15T05:01:00.416 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:01:01.081 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:00 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:00.412377+0000 mon.smithi163 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.178:0/245916470' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:01:01.823 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:01.824 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:02.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:01 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:00.361570+0000 mgr.smithi163.qesyrx (mgr.14162) 169 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:03.081 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.079+0000 7f1a5d269700 1 -- 172.21.15.178:0/2022837983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a580f62a0 msgr2=0x7f1a580f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:03.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.079+0000 7f1a5d269700 1 --2- 172.21.15.178:0/2022837983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a580f62a0 0x7f1a580f66c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f1a4c009fb0 tx=0x7f1a4c009b30).stop 2023-11-15T05:01:03.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.079+0000 7f1a5d269700 1 -- 172.21.15.178:0/2022837983 shutdown_connections 2023-11-15T05:01:03.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.079+0000 7f1a5d269700 1 --2- 172.21.15.178:0/2022837983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a580f62a0 0x7f1a580f66c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:03.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.079+0000 7f1a5d269700 1 -- 172.21.15.178:0/2022837983 >> 172.21.15.178:0/2022837983 conn(0x7f1a580f1aa0 msgr2=0x7f1a580f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:03.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.079+0000 7f1a5d269700 1 -- 172.21.15.178:0/2022837983 shutdown_connections 2023-11-15T05:01:03.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.079+0000 7f1a5d269700 1 -- 172.21.15.178:0/2022837983 wait complete. 2023-11-15T05:01:03.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.080+0000 7f1a5d269700 1 Processor -- start 2023-11-15T05:01:03.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.080+0000 7f1a5d269700 1 -- start start 2023-11-15T05:01:03.084 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.080+0000 7f1a5d269700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a580f62a0 0x7f1a581064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:03.084 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.080+0000 7f1a5d269700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a581069a0 con 0x7f1a580f62a0 2023-11-15T05:01:03.084 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.080+0000 7f1a57fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a580f62a0 0x7f1a581064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:03.084 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.080+0000 7f1a57fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a580f62a0 0x7f1a581064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:38564/0 (socket says 172.21.15.178:38564) 2023-11-15T05:01:03.084 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.080+0000 7f1a57fff700 1 -- 172.21.15.178:0/2823064049 learned_addr learned my addr 172.21.15.178:0/2823064049 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:03.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.081+0000 7f1a57fff700 1 -- 172.21.15.178:0/2823064049 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c005040 con 0x7f1a580f62a0 2023-11-15T05:01:03.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.081+0000 7f1a57fff700 1 --2- 172.21.15.178:0/2823064049 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a580f62a0 0x7f1a581064b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f1a4c013510 tx=0x7f1a4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:03.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.081+0000 7f1a54ff9700 1 -- 172.21.15.178:0/2823064049 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a4c009320 con 0x7f1a580f62a0 2023-11-15T05:01:03.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.081+0000 7f1a54ff9700 1 -- 172.21.15.178:0/2823064049 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a4c013f00 con 0x7f1a580f62a0 2023-11-15T05:01:03.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.081+0000 7f1a5d269700 1 -- 172.21.15.178:0/2823064049 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a58106ba0 con 0x7f1a580f62a0 2023-11-15T05:01:03.086 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.081+0000 7f1a54ff9700 1 -- 172.21.15.178:0/2823064049 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a4c014dc0 con 0x7f1a580f62a0 2023-11-15T05:01:03.086 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.081+0000 7f1a5d269700 1 -- 172.21.15.178:0/2823064049 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a58107030 con 0x7f1a580f62a0 2023-11-15T05:01:03.087 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.082+0000 7f1a54ff9700 1 -- 172.21.15.178:0/2823064049 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1a4c01d3d0 con 0x7f1a580f62a0 2023-11-15T05:01:03.087 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.082+0000 7f1a54ff9700 1 --2- 172.21.15.178:0/2823064049 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1a4002ff60 0x7f1a40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:03.087 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.082+0000 7f1a54ff9700 1 -- 172.21.15.178:0/2823064049 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1a4c0410d0 con 0x7f1a580f62a0 2023-11-15T05:01:03.087 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.082+0000 7f1a5d269700 1 -- 172.21.15.178:0/2823064049 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a58059100 con 0x7f1a580f62a0 2023-11-15T05:01:03.088 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.082+0000 7f1a577fe700 1 --2- 172.21.15.178:0/2823064049 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1a4002ff60 0x7f1a40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:03.088 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.082+0000 7f1a577fe700 1 --2- 172.21.15.178:0/2823064049 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1a4002ff60 0x7f1a40032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f1a480096a0 tx=0x7f1a48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:03.088 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.087+0000 7f1a54ff9700 1 -- 172.21.15.178:0/2823064049 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a4c026490 con 0x7f1a580f62a0 2023-11-15T05:01:03.295 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.294+0000 7f1a5d269700 1 -- 172.21.15.178:0/2823064049 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1a580fbce0 con 0x7f1a580f62a0 2023-11-15T05:01:03.295 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.295+0000 7f1a54ff9700 1 -- 172.21.15.178:0/2823064049 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1a4c0126f0 con 0x7f1a580f62a0 2023-11-15T05:01:03.296 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:03.296 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:03.297 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.297+0000 7f1a5d269700 1 -- 172.21.15.178:0/2823064049 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1a4002ff60 msgr2=0x7f1a40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:03.297 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.297+0000 7f1a5d269700 1 --2- 172.21.15.178:0/2823064049 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1a4002ff60 0x7f1a40032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f1a480096a0 tx=0x7f1a48006b40).stop 2023-11-15T05:01:03.297 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.297+0000 7f1a5d269700 1 -- 172.21.15.178:0/2823064049 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a580f62a0 msgr2=0x7f1a581064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:03.298 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.297+0000 7f1a5d269700 1 --2- 172.21.15.178:0/2823064049 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a580f62a0 0x7f1a581064b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f1a4c013510 tx=0x7f1a4c00b040).stop 2023-11-15T05:01:03.298 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.297+0000 7f1a5d269700 1 -- 172.21.15.178:0/2823064049 shutdown_connections 2023-11-15T05:01:03.298 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.297+0000 7f1a5d269700 1 --2- 172.21.15.178:0/2823064049 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1a4002ff60 0x7f1a40032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:03.298 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.297+0000 7f1a5d269700 1 --2- 172.21.15.178:0/2823064049 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a580f62a0 0x7f1a581064b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:03.298 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.297+0000 7f1a5d269700 1 -- 172.21.15.178:0/2823064049 >> 172.21.15.178:0/2823064049 conn(0x7f1a580f1aa0 msgr2=0x7f1a580f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:03.299 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.297+0000 7f1a5d269700 1 -- 172.21.15.178:0/2823064049 shutdown_connections 2023-11-15T05:01:03.299 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:03.297+0000 7f1a5d269700 1 -- 172.21.15.178:0/2823064049 wait complete. 2023-11-15T05:01:03.299 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:01:04.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:03 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:02.361856+0000 mgr.smithi163.qesyrx (mgr.14162) 170 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:04.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:03 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:03.295826+0000 mon.smithi163 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.178:0/2823064049' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:01:04.698 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:04.699 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:05.921 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.919+0000 7f48b11b6700 1 -- 172.21.15.178:0/1771030782 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f48ac060280 msgr2=0x7f48ac0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:05.922 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.919+0000 7f48b11b6700 1 --2- 172.21.15.178:0/1771030782 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f48ac060280 0x7f48ac0f90f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f489c009fb0 tx=0x7f489c009b30).stop 2023-11-15T05:01:05.922 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.919+0000 7f48b11b6700 1 -- 172.21.15.178:0/1771030782 shutdown_connections 2023-11-15T05:01:05.922 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.919+0000 7f48b11b6700 1 --2- 172.21.15.178:0/1771030782 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f48ac060280 0x7f48ac0f90f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:05.922 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.919+0000 7f48b11b6700 1 -- 172.21.15.178:0/1771030782 >> 172.21.15.178:0/1771030782 conn(0x7f48ac0f0b70 msgr2=0x7f48ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:05.923 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.919+0000 7f48b11b6700 1 -- 172.21.15.178:0/1771030782 shutdown_connections 2023-11-15T05:01:05.923 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.919+0000 7f48b11b6700 1 -- 172.21.15.178:0/1771030782 wait complete. 2023-11-15T05:01:05.923 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.920+0000 7f48b11b6700 1 Processor -- start 2023-11-15T05:01:05.923 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.920+0000 7f48b11b6700 1 -- start start 2023-11-15T05:01:05.923 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.920+0000 7f48b11b6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f48ac060280 0x7f48ac1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:05.923 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.920+0000 7f48b11b6700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ac106bd0 con 0x7f48ac060280 2023-11-15T05:01:05.924 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.920+0000 7f48aad9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f48ac060280 0x7f48ac1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:05.924 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.920+0000 7f48aad9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f48ac060280 0x7f48ac1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:34444/0 (socket says 172.21.15.178:34444) 2023-11-15T05:01:05.924 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.920+0000 7f48aad9d700 1 -- 172.21.15.178:0/2971292075 learned_addr learned my addr 172.21.15.178:0/2971292075 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:05.924 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.921+0000 7f48aad9d700 1 -- 172.21.15.178:0/2971292075 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489c005040 con 0x7f48ac060280 2023-11-15T05:01:05.924 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.921+0000 7f48aad9d700 1 --2- 172.21.15.178:0/2971292075 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f48ac060280 0x7f48ac1066e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f489c009830 tx=0x7f489c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:05.925 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.921+0000 7f48937fe700 1 -- 172.21.15.178:0/2971292075 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f489c009320 con 0x7f48ac060280 2023-11-15T05:01:05.925 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.921+0000 7f48937fe700 1 -- 172.21.15.178:0/2971292075 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f489c0136c0 con 0x7f48ac060280 2023-11-15T05:01:05.925 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.921+0000 7f48b11b6700 1 -- 172.21.15.178:0/2971292075 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48ac106dd0 con 0x7f48ac060280 2023-11-15T05:01:05.926 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.921+0000 7f48937fe700 1 -- 172.21.15.178:0/2971292075 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f489c01ee90 con 0x7f48ac060280 2023-11-15T05:01:05.926 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.921+0000 7f48b11b6700 1 -- 172.21.15.178:0/2971292075 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48ac107260 con 0x7f48ac060280 2023-11-15T05:01:05.927 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.922+0000 7f48937fe700 1 -- 172.21.15.178:0/2971292075 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f489c01e420 con 0x7f48ac060280 2023-11-15T05:01:05.927 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.922+0000 7f48b11b6700 1 -- 172.21.15.178:0/2971292075 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48ac045ad0 con 0x7f48ac060280 2023-11-15T05:01:05.928 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.922+0000 7f48937fe700 1 --2- 172.21.15.178:0/2971292075 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f489402ff10 0x7f48940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:05.928 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.922+0000 7f48937fe700 1 -- 172.21.15.178:0/2971292075 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f489c040f40 con 0x7f48ac060280 2023-11-15T05:01:05.928 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.922+0000 7f48aa59c700 1 --2- 172.21.15.178:0/2971292075 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f489402ff10 0x7f48940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:05.928 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.923+0000 7f48aa59c700 1 --2- 172.21.15.178:0/2971292075 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f489402ff10 0x7f48940323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f48a00096a0 tx=0x7f48a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:05.928 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:05.927+0000 7f48937fe700 1 -- 172.21.15.178:0/2971292075 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f489c06d2b0 con 0x7f48ac060280 2023-11-15T05:01:06.130 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:06.129+0000 7f48b11b6700 1 -- 172.21.15.178:0/2971292075 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48ac108110 con 0x7f48ac060280 2023-11-15T05:01:06.131 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:06.130+0000 7f48937fe700 1 -- 172.21.15.178:0/2971292075 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f489c0095f0 con 0x7f48ac060280 2023-11-15T05:01:06.131 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:06.131 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:06.132 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:06.131+0000 7f48b11b6700 1 -- 172.21.15.178:0/2971292075 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f489402ff10 msgr2=0x7f48940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:06.132 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:06.131+0000 7f48b11b6700 1 --2- 172.21.15.178:0/2971292075 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f489402ff10 0x7f48940323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f48a00096a0 tx=0x7f48a0006b40).stop 2023-11-15T05:01:06.133 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:06.131+0000 7f48b11b6700 1 -- 172.21.15.178:0/2971292075 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f48ac060280 msgr2=0x7f48ac1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:06.133 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:06.131+0000 7f48b11b6700 1 --2- 172.21.15.178:0/2971292075 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f48ac060280 0x7f48ac1066e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f489c009830 tx=0x7f489c00b040).stop 2023-11-15T05:01:06.133 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:06.132+0000 7f48b11b6700 1 -- 172.21.15.178:0/2971292075 shutdown_connections 2023-11-15T05:01:06.133 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:06.132+0000 7f48b11b6700 1 --2- 172.21.15.178:0/2971292075 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f489402ff10 0x7f48940323d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:06.133 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:06.132+0000 7f48b11b6700 1 --2- 172.21.15.178:0/2971292075 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f48ac060280 0x7f48ac1066e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:06.134 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:06.132+0000 7f48b11b6700 1 -- 172.21.15.178:0/2971292075 >> 172.21.15.178:0/2971292075 conn(0x7f48ac0f0b70 msgr2=0x7f48ac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:06.134 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:06.132+0000 7f48b11b6700 1 -- 172.21.15.178:0/2971292075 shutdown_connections 2023-11-15T05:01:06.134 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:06.132+0000 7f48b11b6700 1 -- 172.21.15.178:0/2971292075 wait complete. 2023-11-15T05:01:06.134 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:01:06.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:05 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:04.362156+0000 mgr.smithi163.qesyrx (mgr.14162) 171 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:07.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:06 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:06.130577+0000 mon.smithi163 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.178:0/2971292075' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:01:07.524 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:07.525 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:08.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:07 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:06.362513+0000 mgr.smithi163.qesyrx (mgr.14162) 172 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:08.672 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.670+0000 7f4956aca700 1 -- 172.21.15.178:0/2876477433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49500f5410 msgr2=0x7f49500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:08.673 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.670+0000 7f4956aca700 1 --2- 172.21.15.178:0/2876477433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49500f5410 0x7f49500f5830 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f4944009fb0 tx=0x7f4944009b30).stop 2023-11-15T05:01:08.673 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.670+0000 7f4956aca700 1 -- 172.21.15.178:0/2876477433 shutdown_connections 2023-11-15T05:01:08.673 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.670+0000 7f4956aca700 1 --2- 172.21.15.178:0/2876477433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49500f5410 0x7f49500f5830 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:08.674 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.670+0000 7f4956aca700 1 -- 172.21.15.178:0/2876477433 >> 172.21.15.178:0/2876477433 conn(0x7f49500f0b70 msgr2=0x7f49500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:08.674 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.670+0000 7f4956aca700 1 -- 172.21.15.178:0/2876477433 shutdown_connections 2023-11-15T05:01:08.674 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.670+0000 7f4956aca700 1 -- 172.21.15.178:0/2876477433 wait complete. 2023-11-15T05:01:08.675 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.671+0000 7f4956aca700 1 Processor -- start 2023-11-15T05:01:08.675 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.671+0000 7f4956aca700 1 -- start start 2023-11-15T05:01:08.675 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.671+0000 7f4956aca700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49500f5410 0x7f4950109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:08.675 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.671+0000 7f4956aca700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4950060240 con 0x7f49500f5410 2023-11-15T05:01:08.677 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.672+0000 7f4954866700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49500f5410 0x7f4950109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:08.677 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.672+0000 7f4954866700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49500f5410 0x7f4950109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:34468/0 (socket says 172.21.15.178:34468) 2023-11-15T05:01:08.677 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.672+0000 7f4954866700 1 -- 172.21.15.178:0/557207263 learned_addr learned my addr 172.21.15.178:0/557207263 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:08.677 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.672+0000 7f4954866700 1 -- 172.21.15.178:0/557207263 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4944005040 con 0x7f49500f5410 2023-11-15T05:01:08.678 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.672+0000 7f4954866700 1 --2- 172.21.15.178:0/557207263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49500f5410 0x7f4950109430 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f4944013510 tx=0x7f494400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:08.678 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.672+0000 7f494d7fa700 1 -- 172.21.15.178:0/557207263 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4944009320 con 0x7f49500f5410 2023-11-15T05:01:08.678 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.672+0000 7f4956aca700 1 -- 172.21.15.178:0/557207263 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4950060440 con 0x7f49500f5410 2023-11-15T05:01:08.678 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.672+0000 7f494d7fa700 1 -- 172.21.15.178:0/557207263 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4944013f00 con 0x7f49500f5410 2023-11-15T05:01:08.679 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.672+0000 7f494d7fa700 1 -- 172.21.15.178:0/557207263 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4944014d90 con 0x7f49500f5410 2023-11-15T05:01:08.679 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.672+0000 7f4956aca700 1 -- 172.21.15.178:0/557207263 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4950109ae0 con 0x7f49500f5410 2023-11-15T05:01:08.679 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.673+0000 7f494d7fa700 1 -- 172.21.15.178:0/557207263 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4944014420 con 0x7f49500f5410 2023-11-15T05:01:08.679 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.673+0000 7f4956aca700 1 -- 172.21.15.178:0/557207263 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4950045ad0 con 0x7f49500f5410 2023-11-15T05:01:08.679 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.673+0000 7f494d7fa700 1 --2- 172.21.15.178:0/557207263 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f494002ff60 0x7f4940032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:08.680 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.673+0000 7f494d7fa700 1 -- 172.21.15.178:0/557207263 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f49440410c0 con 0x7f49500f5410 2023-11-15T05:01:08.680 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.673+0000 7f494ffff700 1 --2- 172.21.15.178:0/557207263 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f494002ff60 0x7f4940032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:08.680 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.674+0000 7f494ffff700 1 --2- 172.21.15.178:0/557207263 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f494002ff60 0x7f4940032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f493c0096a0 tx=0x7f493c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:08.680 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.678+0000 7f494d7fa700 1 -- 172.21.15.178:0/557207263 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f49440095f0 con 0x7f49500f5410 2023-11-15T05:01:08.887 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.885+0000 7f4956aca700 1 -- 172.21.15.178:0/557207263 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f49500fae60 con 0x7f49500f5410 2023-11-15T05:01:08.887 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.886+0000 7f494d7fa700 1 -- 172.21.15.178:0/557207263 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f494401db70 con 0x7f49500f5410 2023-11-15T05:01:08.888 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:08.888 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:08.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.888+0000 7f4956aca700 1 -- 172.21.15.178:0/557207263 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f494002ff60 msgr2=0x7f4940032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:08.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.888+0000 7f4956aca700 1 --2- 172.21.15.178:0/557207263 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f494002ff60 0x7f4940032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f493c0096a0 tx=0x7f493c006b40).stop 2023-11-15T05:01:08.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.888+0000 7f4956aca700 1 -- 172.21.15.178:0/557207263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49500f5410 msgr2=0x7f4950109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:08.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.888+0000 7f4956aca700 1 --2- 172.21.15.178:0/557207263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49500f5410 0x7f4950109430 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f4944013510 tx=0x7f494400b040).stop 2023-11-15T05:01:08.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.888+0000 7f4956aca700 1 -- 172.21.15.178:0/557207263 shutdown_connections 2023-11-15T05:01:08.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.888+0000 7f4956aca700 1 --2- 172.21.15.178:0/557207263 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f494002ff60 0x7f4940032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:08.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.888+0000 7f4956aca700 1 --2- 172.21.15.178:0/557207263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49500f5410 0x7f4950109430 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:08.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.888+0000 7f4956aca700 1 -- 172.21.15.178:0/557207263 >> 172.21.15.178:0/557207263 conn(0x7f49500f0b70 msgr2=0x7f49500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:08.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.888+0000 7f4956aca700 1 -- 172.21.15.178:0/557207263 shutdown_connections 2023-11-15T05:01:08.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:08.888+0000 7f4956aca700 1 -- 172.21.15.178:0/557207263 wait complete. 2023-11-15T05:01:08.892 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:01:10.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:09 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:08.362873+0000 mgr.smithi163.qesyrx (mgr.14162) 173 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:10.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:09 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:08.887867+0000 mon.smithi163 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.178:0/557207263' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:01:10.279 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:10.279 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:11.475 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.471+0000 7f74fd1d9700 1 -- 172.21.15.178:0/1758886502 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74f80f51a0 msgr2=0x7f74f80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:11.475 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.471+0000 7f74fd1d9700 1 --2- 172.21.15.178:0/1758886502 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74f80f51a0 0x7f74f80f55c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f74e8009fb0 tx=0x7f74e8009b30).stop 2023-11-15T05:01:11.475 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.471+0000 7f74fd1d9700 1 -- 172.21.15.178:0/1758886502 shutdown_connections 2023-11-15T05:01:11.476 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.471+0000 7f74fd1d9700 1 --2- 172.21.15.178:0/1758886502 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74f80f51a0 0x7f74f80f55c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:11.476 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.471+0000 7f74fd1d9700 1 -- 172.21.15.178:0/1758886502 >> 172.21.15.178:0/1758886502 conn(0x7f74f80f09a0 msgr2=0x7f74f80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:11.476 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.471+0000 7f74fd1d9700 1 -- 172.21.15.178:0/1758886502 shutdown_connections 2023-11-15T05:01:11.477 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.471+0000 7f74fd1d9700 1 -- 172.21.15.178:0/1758886502 wait complete. 2023-11-15T05:01:11.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.471+0000 7f74fd1d9700 1 Processor -- start 2023-11-15T05:01:11.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.471+0000 7f74fd1d9700 1 -- start start 2023-11-15T05:01:11.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.472+0000 7f74fd1d9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74f80f51a0 0x7f74f81086d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:11.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.472+0000 7f74fd1d9700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f8108bc0 con 0x7f74f80f51a0 2023-11-15T05:01:11.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.472+0000 7f74f7fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74f80f51a0 0x7f74f81086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:11.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.472+0000 7f74f7fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74f80f51a0 0x7f74f81086d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:34478/0 (socket says 172.21.15.178:34478) 2023-11-15T05:01:11.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.472+0000 7f74f7fff700 1 -- 172.21.15.178:0/3825503725 learned_addr learned my addr 172.21.15.178:0/3825503725 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:11.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.472+0000 7f74f7fff700 1 -- 172.21.15.178:0/3825503725 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e8005040 con 0x7f74f80f51a0 2023-11-15T05:01:11.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.472+0000 7f74f7fff700 1 --2- 172.21.15.178:0/3825503725 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74f80f51a0 0x7f74f81086d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f74e8013510 tx=0x7f74e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:11.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.473+0000 7f74f57fa700 1 -- 172.21.15.178:0/3825503725 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74e8009320 con 0x7f74f80f51a0 2023-11-15T05:01:11.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.473+0000 7f74fd1d9700 1 -- 172.21.15.178:0/3825503725 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f8108dc0 con 0x7f74f80f51a0 2023-11-15T05:01:11.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.473+0000 7f74f57fa700 1 -- 172.21.15.178:0/3825503725 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f74e8013f00 con 0x7f74f80f51a0 2023-11-15T05:01:11.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.473+0000 7f74fd1d9700 1 -- 172.21.15.178:0/3825503725 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f8109250 con 0x7f74f80f51a0 2023-11-15T05:01:11.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.473+0000 7f74f57fa700 1 -- 172.21.15.178:0/3825503725 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74e8014de0 con 0x7f74f80f51a0 2023-11-15T05:01:11.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.473+0000 7f74f57fa700 1 -- 172.21.15.178:0/3825503725 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f74e8014420 con 0x7f74f80f51a0 2023-11-15T05:01:11.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.473+0000 7f74f57fa700 1 --2- 172.21.15.178:0/3825503725 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f74e402ff60 0x7f74e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:11.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.473+0000 7f74fd1d9700 1 -- 172.21.15.178:0/3825503725 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f8045ad0 con 0x7f74f80f51a0 2023-11-15T05:01:11.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.474+0000 7f74f57fa700 1 -- 172.21.15.178:0/3825503725 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f74e80410a0 con 0x7f74f80f51a0 2023-11-15T05:01:11.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.474+0000 7f74effff700 1 --2- 172.21.15.178:0/3825503725 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f74e402ff60 0x7f74e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:11.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.474+0000 7f74effff700 1 --2- 172.21.15.178:0/3825503725 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f74e402ff60 0x7f74e4032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f74e00096a0 tx=0x7f74e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:11.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.478+0000 7f74f57fa700 1 -- 172.21.15.178:0/3825503725 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74e80095f0 con 0x7f74f80f51a0 2023-11-15T05:01:11.685 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.683+0000 7f74fd1d9700 1 -- 172.21.15.178:0/3825503725 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f74f810a100 con 0x7f74f80f51a0 2023-11-15T05:01:11.685 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.683+0000 7f74f57fa700 1 -- 172.21.15.178:0/3825503725 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f74e801cb10 con 0x7f74f80f51a0 2023-11-15T05:01:11.686 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:11.686 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:11.687 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.685+0000 7f74fd1d9700 1 -- 172.21.15.178:0/3825503725 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f74e402ff60 msgr2=0x7f74e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:11.687 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.685+0000 7f74fd1d9700 1 --2- 172.21.15.178:0/3825503725 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f74e402ff60 0x7f74e4032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f74e00096a0 tx=0x7f74e0006b40).stop 2023-11-15T05:01:11.687 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.685+0000 7f74fd1d9700 1 -- 172.21.15.178:0/3825503725 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74f80f51a0 msgr2=0x7f74f81086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:11.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.685+0000 7f74fd1d9700 1 --2- 172.21.15.178:0/3825503725 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74f80f51a0 0x7f74f81086d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f74e8013510 tx=0x7f74e800b040).stop 2023-11-15T05:01:11.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.685+0000 7f74fd1d9700 1 -- 172.21.15.178:0/3825503725 shutdown_connections 2023-11-15T05:01:11.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.685+0000 7f74fd1d9700 1 --2- 172.21.15.178:0/3825503725 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f74e402ff60 0x7f74e4032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:11.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.685+0000 7f74fd1d9700 1 --2- 172.21.15.178:0/3825503725 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74f80f51a0 0x7f74f81086d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:11.689 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.685+0000 7f74fd1d9700 1 -- 172.21.15.178:0/3825503725 >> 172.21.15.178:0/3825503725 conn(0x7f74f80f09a0 msgr2=0x7f74f805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:11.689 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.685+0000 7f74fd1d9700 1 -- 172.21.15.178:0/3825503725 shutdown_connections 2023-11-15T05:01:11.689 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:11.685+0000 7f74fd1d9700 1 -- 172.21.15.178:0/3825503725 wait complete. 2023-11-15T05:01:11.689 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:01:12.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:11 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:10.363138+0000 mgr.smithi163.qesyrx (mgr.14162) 174 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:12.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:11 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:11.685175+0000 mon.smithi163 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.178:0/3825503725' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:01:13.076 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:13.076 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:14.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:13 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:12.363460+0000 mgr.smithi163.qesyrx (mgr.14162) 175 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:14.250 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.247+0000 7f3bba975700 1 -- 172.21.15.178:0/3156103124 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bb40f47c0 msgr2=0x7f3bb40f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:14.250 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.247+0000 7f3bba975700 1 --2- 172.21.15.178:0/3156103124 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bb40f47c0 0x7f3bb40f4be0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f3ba8009fb0 tx=0x7f3ba8009b30).stop 2023-11-15T05:01:14.250 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.247+0000 7f3bba975700 1 -- 172.21.15.178:0/3156103124 shutdown_connections 2023-11-15T05:01:14.250 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.247+0000 7f3bba975700 1 --2- 172.21.15.178:0/3156103124 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bb40f47c0 0x7f3bb40f4be0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:14.251 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.247+0000 7f3bba975700 1 -- 172.21.15.178:0/3156103124 >> 172.21.15.178:0/3156103124 conn(0x7f3bb40f02e0 msgr2=0x7f3bb40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:14.251 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.248+0000 7f3bba975700 1 -- 172.21.15.178:0/3156103124 shutdown_connections 2023-11-15T05:01:14.251 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.248+0000 7f3bba975700 1 -- 172.21.15.178:0/3156103124 wait complete. 2023-11-15T05:01:14.251 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.248+0000 7f3bba975700 1 Processor -- start 2023-11-15T05:01:14.251 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.248+0000 7f3bba975700 1 -- start start 2023-11-15T05:01:14.252 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.248+0000 7f3bba975700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bb40f47c0 0x7f3bb4108900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:14.252 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.248+0000 7f3bba975700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb4108df0 con 0x7f3bb40f47c0 2023-11-15T05:01:14.252 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.249+0000 7f3bb3fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bb40f47c0 0x7f3bb4108900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:14.252 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.249+0000 7f3bb3fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bb40f47c0 0x7f3bb4108900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:34496/0 (socket says 172.21.15.178:34496) 2023-11-15T05:01:14.252 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.249+0000 7f3bb3fff700 1 -- 172.21.15.178:0/3487416686 learned_addr learned my addr 172.21.15.178:0/3487416686 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:14.254 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.249+0000 7f3bb3fff700 1 -- 172.21.15.178:0/3487416686 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba8005040 con 0x7f3bb40f47c0 2023-11-15T05:01:14.254 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.249+0000 7f3bb3fff700 1 --2- 172.21.15.178:0/3487416686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bb40f47c0 0x7f3bb4108900 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f3ba8013510 tx=0x7f3ba800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:14.254 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.249+0000 7f3bb0ff9700 1 -- 172.21.15.178:0/3487416686 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ba8009320 con 0x7f3bb40f47c0 2023-11-15T05:01:14.255 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.249+0000 7f3bba975700 1 -- 172.21.15.178:0/3487416686 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb4108ff0 con 0x7f3bb40f47c0 2023-11-15T05:01:14.255 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.249+0000 7f3bb0ff9700 1 -- 172.21.15.178:0/3487416686 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ba8013f00 con 0x7f3bb40f47c0 2023-11-15T05:01:14.255 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.249+0000 7f3bb0ff9700 1 -- 172.21.15.178:0/3487416686 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ba8014d90 con 0x7f3bb40f47c0 2023-11-15T05:01:14.255 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.249+0000 7f3bba975700 1 -- 172.21.15.178:0/3487416686 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb4109480 con 0x7f3bb40f47c0 2023-11-15T05:01:14.255 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.250+0000 7f3bb0ff9700 1 -- 172.21.15.178:0/3487416686 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ba80263a0 con 0x7f3bb40f47c0 2023-11-15T05:01:14.256 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.250+0000 7f3bba975700 1 -- 172.21.15.178:0/3487416686 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb4045ad0 con 0x7f3bb40f47c0 2023-11-15T05:01:14.256 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.250+0000 7f3bb0ff9700 1 --2- 172.21.15.178:0/3487416686 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3b9c02ff60 0x7f3b9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:14.256 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.250+0000 7f3bb0ff9700 1 -- 172.21.15.178:0/3487416686 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ba80413a0 con 0x7f3bb40f47c0 2023-11-15T05:01:14.256 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.250+0000 7f3bb37fe700 1 --2- 172.21.15.178:0/3487416686 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3b9c02ff60 0x7f3b9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:14.257 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.251+0000 7f3bb37fe700 1 --2- 172.21.15.178:0/3487416686 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3b9c02ff60 0x7f3b9c032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f3ba40096a0 tx=0x7f3ba4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:14.257 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.255+0000 7f3bb0ff9700 1 -- 172.21.15.178:0/3487416686 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ba8012510 con 0x7f3bb40f47c0 2023-11-15T05:01:14.462 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.460+0000 7f3bba975700 1 -- 172.21.15.178:0/3487416686 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3bb40f6b20 con 0x7f3bb40f47c0 2023-11-15T05:01:14.462 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.460+0000 7f3bb0ff9700 1 -- 172.21.15.178:0/3487416686 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3ba8012330 con 0x7f3bb40f47c0 2023-11-15T05:01:14.463 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:14.463 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:14.464 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.462+0000 7f3bba975700 1 -- 172.21.15.178:0/3487416686 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3b9c02ff60 msgr2=0x7f3b9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:14.464 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.462+0000 7f3bba975700 1 --2- 172.21.15.178:0/3487416686 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3b9c02ff60 0x7f3b9c032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f3ba40096a0 tx=0x7f3ba4006b40).stop 2023-11-15T05:01:14.464 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.462+0000 7f3bba975700 1 -- 172.21.15.178:0/3487416686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bb40f47c0 msgr2=0x7f3bb4108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:14.465 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.462+0000 7f3bba975700 1 --2- 172.21.15.178:0/3487416686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bb40f47c0 0x7f3bb4108900 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f3ba8013510 tx=0x7f3ba800b040).stop 2023-11-15T05:01:14.465 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.462+0000 7f3bba975700 1 -- 172.21.15.178:0/3487416686 shutdown_connections 2023-11-15T05:01:14.465 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.462+0000 7f3bba975700 1 --2- 172.21.15.178:0/3487416686 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3b9c02ff60 0x7f3b9c032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:14.465 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.462+0000 7f3bba975700 1 --2- 172.21.15.178:0/3487416686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bb40f47c0 0x7f3bb4108900 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:14.466 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.462+0000 7f3bba975700 1 -- 172.21.15.178:0/3487416686 >> 172.21.15.178:0/3487416686 conn(0x7f3bb40f02e0 msgr2=0x7f3bb40f10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:14.466 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.462+0000 7f3bba975700 1 -- 172.21.15.178:0/3487416686 shutdown_connections 2023-11-15T05:01:14.466 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:14.462+0000 7f3bba975700 1 -- 172.21.15.178:0/3487416686 wait complete. 2023-11-15T05:01:14.466 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:01:15.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:14 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:14.461929+0000 mon.smithi163 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.178:0/3487416686' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:01:15.852 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:15.852 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:16.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:15 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:14.363722+0000 mgr.smithi163.qesyrx (mgr.14162) 176 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:16.901 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cephadm 2023-11-15T05:01:15.923260+0000 mgr.smithi163.qesyrx (mgr.14162) 177 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi163 ... 2023-11-15T05:01:16.902 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-15T05:01:16.902 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T05:01:16.902 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: stat: stderr time="2023-11-15T04:56:10Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:01:16.903 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: stat: stderr time="2023-11-15T04:56:48Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:01:16.903 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: stat: stderr time="2023-11-15T04:57:39Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:01:16.904 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T05:01:16.904 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-15T05:01:16.904 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T05:01:16.905 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: stat: stderr time="2023-11-15T04:58:53Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:01:16.905 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: stat: stderr time="2023-11-15T04:59:30Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:01:16.905 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: stat: stderr time="2023-11-15T05:00:08Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:01:16.906 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: stat: stderr Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T05:01:16.906 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Traceback (most recent call last): 2023-11-15T05:01:16.906 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T05:01:16.906 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: main() 2023-11-15T05:01:16.906 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T05:01:16.907 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: r = ctx.func(ctx) 2023-11-15T05:01:16.907 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T05:01:16.907 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: return func(ctx) 2023-11-15T05:01:16.907 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-15T05:01:16.908 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-15T05:01:16.908 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-15T05:01:16.908 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-15T05:01:16.908 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-15T05:01:16.908 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: raise RuntimeError('uid/gid not found') 2023-11-15T05:01:16.909 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: RuntimeError: uid/gid not found 2023-11-15T05:01:16.909 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Traceback (most recent call last): 2023-11-15T05:01:16.909 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-15T05:01:16.909 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: yield (conn, connr) 2023-11-15T05:01:16.909 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-15T05:01:16.910 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: code, '\n'.join(err))) 2023-11-15T05:01:16.910 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi163 ... 2023-11-15T05:01:16.910 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-15T05:01:16.910 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T05:01:16.911 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: stat: stderr time="2023-11-15T04:56:10Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:01:16.911 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: stat: stderr time="2023-11-15T04:56:48Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:01:16.911 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: stat: stderr time="2023-11-15T04:57:39Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:01:16.911 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T05:01:16.913 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi163 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-15T05:01:16.913 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T05:01:16.913 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: stat: stderr time="2023-11-15T04:58:53Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:01:16.913 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: stat: stderr time="2023-11-15T04:59:30Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:01:16.914 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: stat: stderr time="2023-11-15T05:00:08Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:01:16.914 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: stat: stderr Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T05:01:16.914 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Traceback (most recent call last): 2023-11-15T05:01:16.915 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T05:01:16.915 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: main() 2023-11-15T05:01:16.915 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T05:01:16.916 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: r = ctx.func(ctx) 2023-11-15T05:01:16.916 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T05:01:16.916 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: return func(ctx) 2023-11-15T05:01:16.916 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-15T05:01:16.917 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-15T05:01:16.917 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-15T05:01:16.917 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-15T05:01:16.917 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-15T05:01:16.917 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: raise RuntimeError('uid/gid not found') 2023-11-15T05:01:16.918 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: RuntimeError: uid/gid not found 2023-11-15T05:01:16.918 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:15.924608+0000 mon.smithi163 (mon.0) 241 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi163", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-15T05:01:16.918 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:15.927172+0000 mon.smithi163 (mon.0) 242 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi163", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-11-15T05:01:16.918 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:15.928016+0000 mon.smithi163 (mon.0) 243 : audit [DBG] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-15T05:01:16.919 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cephadm 2023-11-15T05:01:15.928646+0000 mgr.smithi163.qesyrx (mgr.14162) 178 : cephadm [INF] Deploying daemon crash.smithi163 on smithi163 2023-11-15T05:01:17.006 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.004+0000 7f4d2f59e700 1 -- 172.21.15.178:0/3083486318 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d300f51a0 msgr2=0x7f4d300f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:17.006 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.004+0000 7f4d2f59e700 1 --2- 172.21.15.178:0/3083486318 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d300f51a0 0x7f4d300f55c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f4d20009fb0 tx=0x7f4d20009b30).stop 2023-11-15T05:01:17.006 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.004+0000 7f4d2f59e700 1 -- 172.21.15.178:0/3083486318 shutdown_connections 2023-11-15T05:01:17.006 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.004+0000 7f4d2f59e700 1 --2- 172.21.15.178:0/3083486318 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d300f51a0 0x7f4d300f55c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:17.007 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.004+0000 7f4d2f59e700 1 -- 172.21.15.178:0/3083486318 >> 172.21.15.178:0/3083486318 conn(0x7f4d300f09a0 msgr2=0x7f4d300f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:17.007 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.004+0000 7f4d2f59e700 1 -- 172.21.15.178:0/3083486318 shutdown_connections 2023-11-15T05:01:17.007 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.004+0000 7f4d2f59e700 1 -- 172.21.15.178:0/3083486318 wait complete. 2023-11-15T05:01:17.007 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.004+0000 7f4d2f59e700 1 Processor -- start 2023-11-15T05:01:17.007 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.004+0000 7f4d2f59e700 1 -- start start 2023-11-15T05:01:17.009 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.004+0000 7f4d2f59e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d300f51a0 0x7f4d301064f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:17.009 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.005+0000 7f4d2f59e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d301069e0 con 0x7f4d300f51a0 2023-11-15T05:01:17.009 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.005+0000 7f4d2e59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d300f51a0 0x7f4d301064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:17.010 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.005+0000 7f4d2e59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d300f51a0 0x7f4d301064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:49460/0 (socket says 172.21.15.178:49460) 2023-11-15T05:01:17.010 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.005+0000 7f4d2e59c700 1 -- 172.21.15.178:0/3284685163 learned_addr learned my addr 172.21.15.178:0/3284685163 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:17.010 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.005+0000 7f4d2e59c700 1 -- 172.21.15.178:0/3284685163 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d20005040 con 0x7f4d300f51a0 2023-11-15T05:01:17.010 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.005+0000 7f4d2e59c700 1 --2- 172.21.15.178:0/3284685163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d300f51a0 0x7f4d301064f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f4d20013510 tx=0x7f4d2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:17.010 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.005+0000 7f4d1effd700 1 -- 172.21.15.178:0/3284685163 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d20009320 con 0x7f4d300f51a0 2023-11-15T05:01:17.011 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.005+0000 7f4d1effd700 1 -- 172.21.15.178:0/3284685163 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d20013f00 con 0x7f4d300f51a0 2023-11-15T05:01:17.011 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.005+0000 7f4d2f59e700 1 -- 172.21.15.178:0/3284685163 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d30106be0 con 0x7f4d300f51a0 2023-11-15T05:01:17.011 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.005+0000 7f4d1effd700 1 -- 172.21.15.178:0/3284685163 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d20014dc0 con 0x7f4d300f51a0 2023-11-15T05:01:17.011 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.005+0000 7f4d2f59e700 1 -- 172.21.15.178:0/3284685163 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d30107070 con 0x7f4d300f51a0 2023-11-15T05:01:17.011 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.006+0000 7f4d1effd700 1 -- 172.21.15.178:0/3284685163 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d2001d3d0 con 0x7f4d300f51a0 2023-11-15T05:01:17.012 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.006+0000 7f4d1effd700 1 --2- 172.21.15.178:0/3284685163 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4d1802ff60 0x7f4d18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:17.012 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.006+0000 7f4d1effd700 1 -- 172.21.15.178:0/3284685163 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d20041040 con 0x7f4d300f51a0 2023-11-15T05:01:17.012 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.006+0000 7f4d2f59e700 1 -- 172.21.15.178:0/3284685163 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d30045ad0 con 0x7f4d300f51a0 2023-11-15T05:01:17.012 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.006+0000 7f4d2dd9b700 1 --2- 172.21.15.178:0/3284685163 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4d1802ff60 0x7f4d18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:17.013 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.007+0000 7f4d2dd9b700 1 --2- 172.21.15.178:0/3284685163 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4d1802ff60 0x7f4d18032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f4d240096a0 tx=0x7f4d24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:17.013 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.011+0000 7f4d1effd700 1 -- 172.21.15.178:0/3284685163 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d20012920 con 0x7f4d300f51a0 2023-11-15T05:01:17.222 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.218+0000 7f4d2f59e700 1 -- 172.21.15.178:0/3284685163 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4d30107dc0 con 0x7f4d300f51a0 2023-11-15T05:01:17.225 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.219+0000 7f4d1effd700 1 -- 172.21.15.178:0/3284685163 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4d2002b020 con 0x7f4d300f51a0 2023-11-15T05:01:17.227 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:17.227 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:17.229 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.220+0000 7f4d2f59e700 1 -- 172.21.15.178:0/3284685163 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4d1802ff60 msgr2=0x7f4d18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:17.229 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.220+0000 7f4d2f59e700 1 --2- 172.21.15.178:0/3284685163 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4d1802ff60 0x7f4d18032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f4d240096a0 tx=0x7f4d24006b40).stop 2023-11-15T05:01:17.230 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.220+0000 7f4d2f59e700 1 -- 172.21.15.178:0/3284685163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d300f51a0 msgr2=0x7f4d301064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:17.230 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.220+0000 7f4d2f59e700 1 --2- 172.21.15.178:0/3284685163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d300f51a0 0x7f4d301064f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f4d20013510 tx=0x7f4d2000b040).stop 2023-11-15T05:01:17.231 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.220+0000 7f4d2f59e700 1 -- 172.21.15.178:0/3284685163 shutdown_connections 2023-11-15T05:01:17.231 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.220+0000 7f4d2f59e700 1 --2- 172.21.15.178:0/3284685163 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4d1802ff60 0x7f4d18032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:17.231 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.220+0000 7f4d2f59e700 1 --2- 172.21.15.178:0/3284685163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d300f51a0 0x7f4d301064f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:17.231 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.220+0000 7f4d2f59e700 1 -- 172.21.15.178:0/3284685163 >> 172.21.15.178:0/3284685163 conn(0x7f4d300f09a0 msgr2=0x7f4d300f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:17.231 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.220+0000 7f4d2f59e700 1 -- 172.21.15.178:0/3284685163 shutdown_connections 2023-11-15T05:01:17.232 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:17.220+0000 7f4d2f59e700 1 -- 172.21.15.178:0/3284685163 wait complete. 2023-11-15T05:01:17.232 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:01:18.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:17 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:16.364010+0000 mgr.smithi163.qesyrx (mgr.14162) 179 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:18.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:17 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:17.220441+0000 mon.smithi163 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.178:0/3284685163' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:01:18.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:17 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:17.426084+0000 mon.smithi163 (mon.0) 245 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' 2023-11-15T05:01:18.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:17 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:17.550027+0000 mon.smithi163 (mon.0) 246 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' 2023-11-15T05:01:18.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:17 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:17.553118+0000 mon.smithi163 (mon.0) 247 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' 2023-11-15T05:01:18.204 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:17 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:17.553512+0000 mon.smithi163 (mon.0) 248 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-15T05:01:18.204 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:17 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:17.556946+0000 mon.smithi163 (mon.0) 249 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' 2023-11-15T05:01:18.644 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:18.645 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:19.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:18 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:17.553922+0000 mgr.smithi163.qesyrx (mgr.14162) 180 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-15T05:01:19.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:18 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cephadm 2023-11-15T05:01:17.559821+0000 mgr.smithi163.qesyrx (mgr.14162) 181 : cephadm [INF] Deploying daemon grafana.smithi163 on smithi163 2023-11-15T05:01:19.883 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.880+0000 7f1e87a92700 1 -- 172.21.15.178:0/3212352924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e800f6510 msgr2=0x7f1e800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:19.883 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.880+0000 7f1e87a92700 1 --2- 172.21.15.178:0/3212352924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e800f6510 0x7f1e800f6930 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f1e70009fb0 tx=0x7f1e70009b30).stop 2023-11-15T05:01:19.884 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.881+0000 7f1e87a92700 1 -- 172.21.15.178:0/3212352924 shutdown_connections 2023-11-15T05:01:19.884 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.881+0000 7f1e87a92700 1 --2- 172.21.15.178:0/3212352924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e800f6510 0x7f1e800f6930 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:19.884 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.881+0000 7f1e87a92700 1 -- 172.21.15.178:0/3212352924 >> 172.21.15.178:0/3212352924 conn(0x7f1e800f1c70 msgr2=0x7f1e800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:19.884 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.881+0000 7f1e87a92700 1 -- 172.21.15.178:0/3212352924 shutdown_connections 2023-11-15T05:01:19.884 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.881+0000 7f1e87a92700 1 -- 172.21.15.178:0/3212352924 wait complete. 2023-11-15T05:01:19.885 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.881+0000 7f1e87a92700 1 Processor -- start 2023-11-15T05:01:19.885 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.881+0000 7f1e87a92700 1 -- start start 2023-11-15T05:01:19.885 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.881+0000 7f1e87a92700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e800f6510 0x7f1e80106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:19.885 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.881+0000 7f1e87a92700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e80106c20 con 0x7f1e800f6510 2023-11-15T05:01:19.885 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.882+0000 7f1e8582e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e800f6510 0x7f1e80106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:19.886 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.882+0000 7f1e8582e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e800f6510 0x7f1e80106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:49472/0 (socket says 172.21.15.178:49472) 2023-11-15T05:01:19.886 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.882+0000 7f1e8582e700 1 -- 172.21.15.178:0/3933156132 learned_addr learned my addr 172.21.15.178:0/3933156132 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:19.886 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.882+0000 7f1e8582e700 1 -- 172.21.15.178:0/3933156132 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e70005040 con 0x7f1e800f6510 2023-11-15T05:01:19.886 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.882+0000 7f1e8582e700 1 --2- 172.21.15.178:0/3933156132 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e800f6510 0x7f1e80106730 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f1e70013510 tx=0x7f1e7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:19.886 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.882+0000 7f1e767fc700 1 -- 172.21.15.178:0/3933156132 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e70009320 con 0x7f1e800f6510 2023-11-15T05:01:19.888 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.882+0000 7f1e87a92700 1 -- 172.21.15.178:0/3933156132 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e80106e20 con 0x7f1e800f6510 2023-11-15T05:01:19.888 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.882+0000 7f1e767fc700 1 -- 172.21.15.178:0/3933156132 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e70013f00 con 0x7f1e800f6510 2023-11-15T05:01:19.888 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.882+0000 7f1e767fc700 1 -- 172.21.15.178:0/3933156132 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e70014d90 con 0x7f1e800f6510 2023-11-15T05:01:19.889 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.882+0000 7f1e87a92700 1 -- 172.21.15.178:0/3933156132 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e801072b0 con 0x7f1e800f6510 2023-11-15T05:01:19.889 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.883+0000 7f1e767fc700 1 -- 172.21.15.178:0/3933156132 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e700095f0 con 0x7f1e800f6510 2023-11-15T05:01:19.889 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.883+0000 7f1e87a92700 1 -- 172.21.15.178:0/3933156132 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e80045ad0 con 0x7f1e800f6510 2023-11-15T05:01:19.889 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.883+0000 7f1e767fc700 1 --2- 172.21.15.178:0/3933156132 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1e6c02ff60 0x7f1e6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:19.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.883+0000 7f1e767fc700 1 -- 172.21.15.178:0/3933156132 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e700410c0 con 0x7f1e800f6510 2023-11-15T05:01:19.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.883+0000 7f1e8502d700 1 --2- 172.21.15.178:0/3933156132 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1e6c02ff60 0x7f1e6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:19.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.884+0000 7f1e8502d700 1 --2- 172.21.15.178:0/3933156132 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1e6c02ff60 0x7f1e6c032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f1e7c0096a0 tx=0x7f1e7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:19.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:19.888+0000 7f1e767fc700 1 -- 172.21.15.178:0/3933156132 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e70014420 con 0x7f1e800f6510 2023-11-15T05:01:20.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:20.093+0000 7f1e87a92700 1 -- 172.21.15.178:0/3933156132 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e800fbf60 con 0x7f1e800f6510 2023-11-15T05:01:20.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:20.094+0000 7f1e767fc700 1 -- 172.21.15.178:0/3933156132 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1e70010230 con 0x7f1e800f6510 2023-11-15T05:01:20.095 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:20.095 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:20.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:20.095+0000 7f1e87a92700 1 -- 172.21.15.178:0/3933156132 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1e6c02ff60 msgr2=0x7f1e6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:20.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:20.095+0000 7f1e87a92700 1 --2- 172.21.15.178:0/3933156132 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1e6c02ff60 0x7f1e6c032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f1e7c0096a0 tx=0x7f1e7c006b40).stop 2023-11-15T05:01:20.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:20.095+0000 7f1e87a92700 1 -- 172.21.15.178:0/3933156132 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e800f6510 msgr2=0x7f1e80106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:20.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:20.095+0000 7f1e87a92700 1 --2- 172.21.15.178:0/3933156132 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e800f6510 0x7f1e80106730 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f1e70013510 tx=0x7f1e7000b040).stop 2023-11-15T05:01:20.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:20.095+0000 7f1e87a92700 1 -- 172.21.15.178:0/3933156132 shutdown_connections 2023-11-15T05:01:20.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:20.095+0000 7f1e87a92700 1 --2- 172.21.15.178:0/3933156132 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1e6c02ff60 0x7f1e6c032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:20.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:20.095+0000 7f1e87a92700 1 --2- 172.21.15.178:0/3933156132 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e800f6510 0x7f1e80106730 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:20.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:20.095+0000 7f1e87a92700 1 -- 172.21.15.178:0/3933156132 >> 172.21.15.178:0/3933156132 conn(0x7f1e800f1c70 msgr2=0x7f1e800f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:20.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:20.095+0000 7f1e87a92700 1 -- 172.21.15.178:0/3933156132 shutdown_connections 2023-11-15T05:01:20.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:20.095+0000 7f1e87a92700 1 -- 172.21.15.178:0/3933156132 wait complete. 2023-11-15T05:01:20.099 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:01:20.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:19 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:18.364267+0000 mgr.smithi163.qesyrx (mgr.14162) 182 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:21.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:20 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:20.095285+0000 mon.smithi163 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.178:0/3933156132' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:01:21.497 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:21.498 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:22.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:21 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:20.364566+0000 mgr.smithi163.qesyrx (mgr.14162) 183 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:22.687 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.685+0000 7f0b46153700 1 -- 172.21.15.178:0/3856646955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b400601e0 msgr2=0x7f0b400f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:22.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.685+0000 7f0b46153700 1 --2- 172.21.15.178:0/3856646955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b400601e0 0x7f0b400f8f20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f0b3c009fb0 tx=0x7f0b3c009b30).stop 2023-11-15T05:01:22.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.685+0000 7f0b46153700 1 -- 172.21.15.178:0/3856646955 shutdown_connections 2023-11-15T05:01:22.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.685+0000 7f0b46153700 1 --2- 172.21.15.178:0/3856646955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b400601e0 0x7f0b400f8f20 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:22.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.685+0000 7f0b46153700 1 -- 172.21.15.178:0/3856646955 >> 172.21.15.178:0/3856646955 conn(0x7f0b400f09a0 msgr2=0x7f0b400f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:22.688 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.685+0000 7f0b46153700 1 -- 172.21.15.178:0/3856646955 shutdown_connections 2023-11-15T05:01:22.689 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.685+0000 7f0b46153700 1 -- 172.21.15.178:0/3856646955 wait complete. 2023-11-15T05:01:22.689 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.686+0000 7f0b46153700 1 Processor -- start 2023-11-15T05:01:22.689 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.686+0000 7f0b46153700 1 -- start start 2023-11-15T05:01:22.689 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.686+0000 7f0b46153700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b400601e0 0x7f0b40106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:22.689 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.686+0000 7f0b46153700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b401069f0 con 0x7f0b400601e0 2023-11-15T05:01:22.690 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.686+0000 7f0b45151700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b400601e0 0x7f0b40106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:22.690 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.686+0000 7f0b45151700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b400601e0 0x7f0b40106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:49476/0 (socket says 172.21.15.178:49476) 2023-11-15T05:01:22.690 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.686+0000 7f0b45151700 1 -- 172.21.15.178:0/2009150617 learned_addr learned my addr 172.21.15.178:0/2009150617 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:22.690 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.686+0000 7f0b45151700 1 -- 172.21.15.178:0/2009150617 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3c005040 con 0x7f0b400601e0 2023-11-15T05:01:22.692 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.687+0000 7f0b45151700 1 --2- 172.21.15.178:0/2009150617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b400601e0 0x7f0b40106500 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f0b3c013510 tx=0x7f0b3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:22.692 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.687+0000 7f0b35ffb700 1 -- 172.21.15.178:0/2009150617 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b3c009320 con 0x7f0b400601e0 2023-11-15T05:01:22.692 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.687+0000 7f0b35ffb700 1 -- 172.21.15.178:0/2009150617 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b3c013f00 con 0x7f0b400601e0 2023-11-15T05:01:22.692 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.687+0000 7f0b35ffb700 1 -- 172.21.15.178:0/2009150617 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b3c014d90 con 0x7f0b400601e0 2023-11-15T05:01:22.692 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.687+0000 7f0b46153700 1 -- 172.21.15.178:0/2009150617 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b40106bf0 con 0x7f0b400601e0 2023-11-15T05:01:22.693 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.687+0000 7f0b46153700 1 -- 172.21.15.178:0/2009150617 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b40107080 con 0x7f0b400601e0 2023-11-15T05:01:22.693 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.688+0000 7f0b35ffb700 1 -- 172.21.15.178:0/2009150617 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0b3c0095f0 con 0x7f0b400601e0 2023-11-15T05:01:22.693 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.688+0000 7f0b46153700 1 -- 172.21.15.178:0/2009150617 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b40045ad0 con 0x7f0b400601e0 2023-11-15T05:01:22.693 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.688+0000 7f0b35ffb700 1 --2- 172.21.15.178:0/2009150617 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f0b2c02ff60 0x7f0b2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:22.693 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.688+0000 7f0b35ffb700 1 -- 172.21.15.178:0/2009150617 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0b3c041000 con 0x7f0b400601e0 2023-11-15T05:01:22.694 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.688+0000 7f0b44950700 1 --2- 172.21.15.178:0/2009150617 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f0b2c02ff60 0x7f0b2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:22.694 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.689+0000 7f0b44950700 1 --2- 172.21.15.178:0/2009150617 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f0b2c02ff60 0x7f0b2c032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f0b300096a0 tx=0x7f0b30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:22.694 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.693+0000 7f0b35ffb700 1 -- 172.21.15.178:0/2009150617 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b3c01d7f0 con 0x7f0b400601e0 2023-11-15T05:01:22.901 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.900+0000 7f0b46153700 1 -- 172.21.15.178:0/2009150617 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b40107f30 con 0x7f0b400601e0 2023-11-15T05:01:22.902 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.901+0000 7f0b35ffb700 1 -- 172.21.15.178:0/2009150617 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0b3c010230 con 0x7f0b400601e0 2023-11-15T05:01:22.902 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:22.903 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:22.904 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.902+0000 7f0b46153700 1 -- 172.21.15.178:0/2009150617 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f0b2c02ff60 msgr2=0x7f0b2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:22.904 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.902+0000 7f0b46153700 1 --2- 172.21.15.178:0/2009150617 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f0b2c02ff60 0x7f0b2c032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f0b300096a0 tx=0x7f0b30006b40).stop 2023-11-15T05:01:22.904 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.902+0000 7f0b46153700 1 -- 172.21.15.178:0/2009150617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b400601e0 msgr2=0x7f0b40106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:22.904 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.902+0000 7f0b46153700 1 --2- 172.21.15.178:0/2009150617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b400601e0 0x7f0b40106500 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f0b3c013510 tx=0x7f0b3c00b040).stop 2023-11-15T05:01:22.904 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.902+0000 7f0b46153700 1 -- 172.21.15.178:0/2009150617 shutdown_connections 2023-11-15T05:01:22.905 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.902+0000 7f0b46153700 1 --2- 172.21.15.178:0/2009150617 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f0b2c02ff60 0x7f0b2c032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:22.905 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.902+0000 7f0b46153700 1 --2- 172.21.15.178:0/2009150617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b400601e0 0x7f0b40106500 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:22.905 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.902+0000 7f0b46153700 1 -- 172.21.15.178:0/2009150617 >> 172.21.15.178:0/2009150617 conn(0x7f0b400f09a0 msgr2=0x7f0b400f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:22.905 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.902+0000 7f0b46153700 1 -- 172.21.15.178:0/2009150617 shutdown_connections 2023-11-15T05:01:22.905 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:22.902+0000 7f0b46153700 1 -- 172.21.15.178:0/2009150617 wait complete. 2023-11-15T05:01:22.906 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:01:23.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:22 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:22.902450+0000 mon.smithi163 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.178:0/2009150617' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:01:24.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:23 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:22.364818+0000 mgr.smithi163.qesyrx (mgr.14162) 184 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:24.286 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:24.286 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:25.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:24 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:24.453612+0000 mon.smithi163 (mon.0) 252 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:01:25.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:24 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:24.454975+0000 mon.smithi163 (mon.0) 253 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/trash_purge_schedule"}]: dispatch 2023-11-15T05:01:25.537 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.536+0000 7fcda1f03700 1 -- 172.21.15.178:0/2399754667 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd9c0f5410 msgr2=0x7fcd9c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:25.538 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.536+0000 7fcda1f03700 1 --2- 172.21.15.178:0/2399754667 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd9c0f5410 0x7fcd9c0f5830 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fcd90009fb0 tx=0x7fcd90009b30).stop 2023-11-15T05:01:25.538 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.536+0000 7fcda1f03700 1 -- 172.21.15.178:0/2399754667 shutdown_connections 2023-11-15T05:01:25.538 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.536+0000 7fcda1f03700 1 --2- 172.21.15.178:0/2399754667 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd9c0f5410 0x7fcd9c0f5830 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:25.539 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.536+0000 7fcda1f03700 1 -- 172.21.15.178:0/2399754667 >> 172.21.15.178:0/2399754667 conn(0x7fcd9c0f0b70 msgr2=0x7fcd9c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:25.539 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.536+0000 7fcda1f03700 1 -- 172.21.15.178:0/2399754667 shutdown_connections 2023-11-15T05:01:25.539 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.536+0000 7fcda1f03700 1 -- 172.21.15.178:0/2399754667 wait complete. 2023-11-15T05:01:25.542 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.537+0000 7fcda1f03700 1 Processor -- start 2023-11-15T05:01:25.542 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.537+0000 7fcda1f03700 1 -- start start 2023-11-15T05:01:25.542 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.537+0000 7fcda1f03700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd9c0f5410 0x7fcd9c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:25.542 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.537+0000 7fcda1f03700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c105b20 con 0x7fcd9c0f5410 2023-11-15T05:01:25.543 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.537+0000 7fcd9b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd9c0f5410 0x7fcd9c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:25.543 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.537+0000 7fcd9b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd9c0f5410 0x7fcd9c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:56954/0 (socket says 172.21.15.178:56954) 2023-11-15T05:01:25.543 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.537+0000 7fcd9b7fe700 1 -- 172.21.15.178:0/2031108867 learned_addr learned my addr 172.21.15.178:0/2031108867 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:25.543 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.538+0000 7fcd9b7fe700 1 -- 172.21.15.178:0/2031108867 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd90005040 con 0x7fcd9c0f5410 2023-11-15T05:01:25.543 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.538+0000 7fcd9b7fe700 1 --2- 172.21.15.178:0/2031108867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd9c0f5410 0x7fcd9c105630 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fcd90014000 tx=0x7fcd9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:25.544 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.538+0000 7fcd83fff700 1 -- 172.21.15.178:0/2031108867 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd900095f0 con 0x7fcd9c0f5410 2023-11-15T05:01:25.544 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.538+0000 7fcd83fff700 1 -- 172.21.15.178:0/2031108867 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd90007e70 con 0x7fcd9c0f5410 2023-11-15T05:01:25.545 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.538+0000 7fcda1f03700 1 -- 172.21.15.178:0/2031108867 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9c105d20 con 0x7fcd9c0f5410 2023-11-15T05:01:25.545 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.538+0000 7fcd83fff700 1 -- 172.21.15.178:0/2031108867 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd90014ea0 con 0x7fcd9c0f5410 2023-11-15T05:01:25.545 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.538+0000 7fcda1f03700 1 -- 172.21.15.178:0/2031108867 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9c1061b0 con 0x7fcd9c0f5410 2023-11-15T05:01:25.546 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.539+0000 7fcd83fff700 1 -- 172.21.15.178:0/2031108867 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcd90025070 con 0x7fcd9c0f5410 2023-11-15T05:01:25.546 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.539+0000 7fcd83fff700 1 --2- 172.21.15.178:0/2031108867 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcd8402ff60 0x7fcd84032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:25.547 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.539+0000 7fcda1f03700 1 -- 172.21.15.178:0/2031108867 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9c045ad0 con 0x7fcd9c0f5410 2023-11-15T05:01:25.547 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.539+0000 7fcd83fff700 1 -- 172.21.15.178:0/2031108867 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcd90040c80 con 0x7fcd9c0f5410 2023-11-15T05:01:25.547 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.539+0000 7fcd9affd700 1 --2- 172.21.15.178:0/2031108867 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcd8402ff60 0x7fcd84032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:25.547 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.539+0000 7fcd9affd700 1 --2- 172.21.15.178:0/2031108867 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcd8402ff60 0x7fcd84032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fcd8c0096a0 tx=0x7fcd8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:25.548 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.543+0000 7fcd83fff700 1 -- 172.21.15.178:0/2031108867 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd90025950 con 0x7fcd9c0f5410 2023-11-15T05:01:25.748 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.747+0000 7fcda1f03700 1 -- 172.21.15.178:0/2031108867 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd9c0faf50 con 0x7fcd9c0f5410 2023-11-15T05:01:25.749 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.747+0000 7fcd83fff700 1 -- 172.21.15.178:0/2031108867 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcd90025950 con 0x7fcd9c0f5410 2023-11-15T05:01:25.749 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:25.750 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:25.750 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.749+0000 7fcda1f03700 1 -- 172.21.15.178:0/2031108867 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcd8402ff60 msgr2=0x7fcd84032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:25.751 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.749+0000 7fcda1f03700 1 --2- 172.21.15.178:0/2031108867 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcd8402ff60 0x7fcd84032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fcd8c0096a0 tx=0x7fcd8c006b40).stop 2023-11-15T05:01:25.751 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.749+0000 7fcda1f03700 1 -- 172.21.15.178:0/2031108867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd9c0f5410 msgr2=0x7fcd9c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:25.751 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.749+0000 7fcda1f03700 1 --2- 172.21.15.178:0/2031108867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd9c0f5410 0x7fcd9c105630 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fcd90014000 tx=0x7fcd9000b040).stop 2023-11-15T05:01:25.751 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.749+0000 7fcda1f03700 1 -- 172.21.15.178:0/2031108867 shutdown_connections 2023-11-15T05:01:25.752 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.749+0000 7fcda1f03700 1 --2- 172.21.15.178:0/2031108867 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcd8402ff60 0x7fcd84032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:25.752 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.749+0000 7fcda1f03700 1 --2- 172.21.15.178:0/2031108867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd9c0f5410 0x7fcd9c105630 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:25.752 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.749+0000 7fcda1f03700 1 -- 172.21.15.178:0/2031108867 >> 172.21.15.178:0/2031108867 conn(0x7fcd9c0f0b70 msgr2=0x7fcd9c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:25.752 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.749+0000 7fcda1f03700 1 -- 172.21.15.178:0/2031108867 shutdown_connections 2023-11-15T05:01:25.752 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:25.749+0000 7fcda1f03700 1 -- 172.21.15.178:0/2031108867 wait complete. 2023-11-15T05:01:25.752 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:01:26.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:24.365194+0000 mgr.smithi163.qesyrx (mgr.14162) 185 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:26.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:25.748894+0000 mon.smithi163 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.178:0/2031108867' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:01:27.134 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:27.135 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:28.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:27 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:26.365421+0000 mgr.smithi163.qesyrx (mgr.14162) 186 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:28.273 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.269+0000 7f65aac32700 1 -- 172.21.15.178:0/947968859 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65a40f51a0 msgr2=0x7f65a40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:28.273 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.269+0000 7f65aac32700 1 --2- 172.21.15.178:0/947968859 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65a40f51a0 0x7f65a40f55c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f6594009fb0 tx=0x7f6594009b30).stop 2023-11-15T05:01:28.274 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.269+0000 7f65aac32700 1 -- 172.21.15.178:0/947968859 shutdown_connections 2023-11-15T05:01:28.274 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.269+0000 7f65aac32700 1 --2- 172.21.15.178:0/947968859 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65a40f51a0 0x7f65a40f55c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:28.275 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.269+0000 7f65aac32700 1 -- 172.21.15.178:0/947968859 >> 172.21.15.178:0/947968859 conn(0x7f65a40f09a0 msgr2=0x7f65a40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:28.275 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.269+0000 7f65aac32700 1 -- 172.21.15.178:0/947968859 shutdown_connections 2023-11-15T05:01:28.275 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.269+0000 7f65aac32700 1 -- 172.21.15.178:0/947968859 wait complete. 2023-11-15T05:01:28.275 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.270+0000 7f65aac32700 1 Processor -- start 2023-11-15T05:01:28.276 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.270+0000 7f65aac32700 1 -- start start 2023-11-15T05:01:28.276 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.270+0000 7f65aac32700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65a40f51a0 0x7f65a41064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:28.276 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.270+0000 7f65aac32700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a41069d0 con 0x7f65a40f51a0 2023-11-15T05:01:28.278 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.270+0000 7f65a9c30700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65a40f51a0 0x7f65a41064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:28.278 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.270+0000 7f65a9c30700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65a40f51a0 0x7f65a41064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:56978/0 (socket says 172.21.15.178:56978) 2023-11-15T05:01:28.278 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.270+0000 7f65a9c30700 1 -- 172.21.15.178:0/3440099349 learned_addr learned my addr 172.21.15.178:0/3440099349 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:28.279 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.270+0000 7f65a9c30700 1 -- 172.21.15.178:0/3440099349 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6594005040 con 0x7f65a40f51a0 2023-11-15T05:01:28.279 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.271+0000 7f65a9c30700 1 --2- 172.21.15.178:0/3440099349 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65a40f51a0 0x7f65a41064e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f6594013510 tx=0x7f659400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:28.279 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.271+0000 7f659a7fc700 1 -- 172.21.15.178:0/3440099349 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6594009320 con 0x7f65a40f51a0 2023-11-15T05:01:28.279 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.271+0000 7f65aac32700 1 -- 172.21.15.178:0/3440099349 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65a4106bd0 con 0x7f65a40f51a0 2023-11-15T05:01:28.279 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.271+0000 7f659a7fc700 1 -- 172.21.15.178:0/3440099349 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6594013f00 con 0x7f65a40f51a0 2023-11-15T05:01:28.280 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.271+0000 7f659a7fc700 1 -- 172.21.15.178:0/3440099349 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6594014dc0 con 0x7f65a40f51a0 2023-11-15T05:01:28.280 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.271+0000 7f65aac32700 1 -- 172.21.15.178:0/3440099349 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65a4107060 con 0x7f65a40f51a0 2023-11-15T05:01:28.280 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.271+0000 7f659a7fc700 1 -- 172.21.15.178:0/3440099349 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f659401d3d0 con 0x7f65a40f51a0 2023-11-15T05:01:28.280 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.272+0000 7f659a7fc700 1 --2- 172.21.15.178:0/3440099349 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f659002ff60 0x7f6590032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:28.281 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.272+0000 7f659a7fc700 1 -- 172.21.15.178:0/3440099349 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6594041030 con 0x7f65a40f51a0 2023-11-15T05:01:28.281 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.272+0000 7f65aac32700 1 -- 172.21.15.178:0/3440099349 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65a4059100 con 0x7f65a40f51a0 2023-11-15T05:01:28.281 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.272+0000 7f65a942f700 1 --2- 172.21.15.178:0/3440099349 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f659002ff60 0x7f6590032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:28.281 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.272+0000 7f65a942f700 1 --2- 172.21.15.178:0/3440099349 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f659002ff60 0x7f6590032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f65a00096a0 tx=0x7f65a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:28.281 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.277+0000 7f659a7fc700 1 -- 172.21.15.178:0/3440099349 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6594012920 con 0x7f65a40f51a0 2023-11-15T05:01:28.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.484+0000 7f65aac32700 1 -- 172.21.15.178:0/3440099349 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f65a40fabe0 con 0x7f65a40f51a0 2023-11-15T05:01:28.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.485+0000 7f659a7fc700 1 -- 172.21.15.178:0/3440099349 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6594012330 con 0x7f65a40f51a0 2023-11-15T05:01:28.487 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:28.487 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:28.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.486+0000 7f65aac32700 1 -- 172.21.15.178:0/3440099349 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f659002ff60 msgr2=0x7f6590032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:28.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.486+0000 7f65aac32700 1 --2- 172.21.15.178:0/3440099349 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f659002ff60 0x7f6590032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f65a00096a0 tx=0x7f65a0006b40).stop 2023-11-15T05:01:28.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.486+0000 7f65aac32700 1 -- 172.21.15.178:0/3440099349 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65a40f51a0 msgr2=0x7f65a41064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:28.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.486+0000 7f65aac32700 1 --2- 172.21.15.178:0/3440099349 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65a40f51a0 0x7f65a41064e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f6594013510 tx=0x7f659400b040).stop 2023-11-15T05:01:28.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.487+0000 7f65aac32700 1 -- 172.21.15.178:0/3440099349 shutdown_connections 2023-11-15T05:01:28.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.487+0000 7f65aac32700 1 --2- 172.21.15.178:0/3440099349 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f659002ff60 0x7f6590032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:28.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.487+0000 7f65aac32700 1 --2- 172.21.15.178:0/3440099349 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65a40f51a0 0x7f65a41064e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:28.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.487+0000 7f65aac32700 1 -- 172.21.15.178:0/3440099349 >> 172.21.15.178:0/3440099349 conn(0x7f65a40f09a0 msgr2=0x7f65a405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:28.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.487+0000 7f65aac32700 1 -- 172.21.15.178:0/3440099349 shutdown_connections 2023-11-15T05:01:28.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:28.487+0000 7f65aac32700 1 -- 172.21.15.178:0/3440099349 wait complete. 2023-11-15T05:01:28.490 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:01:29.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:28 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:28.486646+0000 mon.smithi163 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.178:0/3440099349' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:01:29.879 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:29.880 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:30.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:29 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:28.365624+0000 mgr.smithi163.qesyrx (mgr.14162) 187 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:31.064 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.062+0000 7fdfea331700 1 -- 172.21.15.178:0/2036939515 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdfe40f6510 msgr2=0x7fdfe40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:31.064 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.062+0000 7fdfea331700 1 --2- 172.21.15.178:0/2036939515 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdfe40f6510 0x7fdfe40f6930 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fdfd4009fb0 tx=0x7fdfd4009b30).stop 2023-11-15T05:01:31.064 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.062+0000 7fdfea331700 1 -- 172.21.15.178:0/2036939515 shutdown_connections 2023-11-15T05:01:31.065 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.062+0000 7fdfea331700 1 --2- 172.21.15.178:0/2036939515 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdfe40f6510 0x7fdfe40f6930 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:31.065 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.062+0000 7fdfea331700 1 -- 172.21.15.178:0/2036939515 >> 172.21.15.178:0/2036939515 conn(0x7fdfe40f1c70 msgr2=0x7fdfe40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:31.065 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.062+0000 7fdfea331700 1 -- 172.21.15.178:0/2036939515 shutdown_connections 2023-11-15T05:01:31.065 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.062+0000 7fdfea331700 1 -- 172.21.15.178:0/2036939515 wait complete. 2023-11-15T05:01:31.065 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.063+0000 7fdfea331700 1 Processor -- start 2023-11-15T05:01:31.066 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.063+0000 7fdfea331700 1 -- start start 2023-11-15T05:01:31.066 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.063+0000 7fdfea331700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdfe40f6510 0x7fdfe406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:31.066 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.063+0000 7fdfea331700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe4069e20 con 0x7fdfe40f6510 2023-11-15T05:01:31.066 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.063+0000 7fdfe3fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdfe40f6510 0x7fdfe406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:31.066 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.063+0000 7fdfe3fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdfe40f6510 0x7fdfe406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:56990/0 (socket says 172.21.15.178:56990) 2023-11-15T05:01:31.067 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.063+0000 7fdfe3fff700 1 -- 172.21.15.178:0/550944997 learned_addr learned my addr 172.21.15.178:0/550944997 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:31.067 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.063+0000 7fdfe3fff700 1 -- 172.21.15.178:0/550944997 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd4005040 con 0x7fdfe40f6510 2023-11-15T05:01:31.067 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.063+0000 7fdfe3fff700 1 --2- 172.21.15.178:0/550944997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdfe40f6510 0x7fdfe406b9a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fdfd4013510 tx=0x7fdfd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:31.067 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.064+0000 7fdfe0ff9700 1 -- 172.21.15.178:0/550944997 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfd4009320 con 0x7fdfe40f6510 2023-11-15T05:01:31.068 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.064+0000 7fdfea331700 1 -- 172.21.15.178:0/550944997 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe4069fc0 con 0x7fdfe40f6510 2023-11-15T05:01:31.069 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.064+0000 7fdfe0ff9700 1 -- 172.21.15.178:0/550944997 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfd4013f00 con 0x7fdfe40f6510 2023-11-15T05:01:31.069 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.064+0000 7fdfe0ff9700 1 -- 172.21.15.178:0/550944997 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfd4014d90 con 0x7fdfe40f6510 2023-11-15T05:01:31.069 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.064+0000 7fdfea331700 1 -- 172.21.15.178:0/550944997 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe406a450 con 0x7fdfe40f6510 2023-11-15T05:01:31.069 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.064+0000 7fdfe0ff9700 1 -- 172.21.15.178:0/550944997 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdfd4014420 con 0x7fdfe40f6510 2023-11-15T05:01:31.069 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.064+0000 7fdfea331700 1 -- 172.21.15.178:0/550944997 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe4059100 con 0x7fdfe40f6510 2023-11-15T05:01:31.070 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.065+0000 7fdfe0ff9700 1 --2- 172.21.15.178:0/550944997 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdfcc02ff60 0x7fdfcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:31.070 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.065+0000 7fdfe0ff9700 1 -- 172.21.15.178:0/550944997 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdfd4041000 con 0x7fdfe40f6510 2023-11-15T05:01:31.070 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.065+0000 7fdfe37fe700 1 --2- 172.21.15.178:0/550944997 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdfcc02ff60 0x7fdfcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:31.070 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.065+0000 7fdfe37fe700 1 --2- 172.21.15.178:0/550944997 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdfcc02ff60 0x7fdfcc032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fdfd80096a0 tx=0x7fdfd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:31.071 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.069+0000 7fdfe0ff9700 1 -- 172.21.15.178:0/550944997 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdfd40095f0 con 0x7fdfe40f6510 2023-11-15T05:01:31.276 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.274+0000 7fdfea331700 1 -- 172.21.15.178:0/550944997 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdfe40fbf60 con 0x7fdfe40f6510 2023-11-15T05:01:31.276 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.275+0000 7fdfe0ff9700 1 -- 172.21.15.178:0/550944997 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdfd401db70 con 0x7fdfe40f6510 2023-11-15T05:01:31.277 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:31.277 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:31.278 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.277+0000 7fdfea331700 1 -- 172.21.15.178:0/550944997 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdfcc02ff60 msgr2=0x7fdfcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:31.278 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.277+0000 7fdfea331700 1 --2- 172.21.15.178:0/550944997 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdfcc02ff60 0x7fdfcc032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fdfd80096a0 tx=0x7fdfd8006b40).stop 2023-11-15T05:01:31.278 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.277+0000 7fdfea331700 1 -- 172.21.15.178:0/550944997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdfe40f6510 msgr2=0x7fdfe406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:31.279 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.277+0000 7fdfea331700 1 --2- 172.21.15.178:0/550944997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdfe40f6510 0x7fdfe406b9a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fdfd4013510 tx=0x7fdfd400b040).stop 2023-11-15T05:01:31.279 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.277+0000 7fdfea331700 1 -- 172.21.15.178:0/550944997 shutdown_connections 2023-11-15T05:01:31.279 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.277+0000 7fdfea331700 1 --2- 172.21.15.178:0/550944997 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdfcc02ff60 0x7fdfcc032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:31.279 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.277+0000 7fdfea331700 1 --2- 172.21.15.178:0/550944997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdfe40f6510 0x7fdfe406b9a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:31.279 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.277+0000 7fdfea331700 1 -- 172.21.15.178:0/550944997 >> 172.21.15.178:0/550944997 conn(0x7fdfe40f1c70 msgr2=0x7fdfe40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:31.279 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.277+0000 7fdfea331700 1 -- 172.21.15.178:0/550944997 shutdown_connections 2023-11-15T05:01:31.280 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:31.277+0000 7fdfea331700 1 -- 172.21.15.178:0/550944997 wait complete. 2023-11-15T05:01:31.280 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:01:32.212 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:31 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:30.365825+0000 mgr.smithi163.qesyrx (mgr.14162) 188 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:32.212 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:31 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:31.275790+0000 mon.smithi163 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.178:0/550944997' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:01:32.662 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:32.663 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:33.962 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.961+0000 7f02a73f2700 1 -- 172.21.15.178:0/1221002274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f02a00f78e0 msgr2=0x7f02a00f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:33.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.961+0000 7f02a73f2700 1 --2- 172.21.15.178:0/1221002274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f02a00f78e0 0x7f02a00f7d00 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f029c009fb0 tx=0x7f029c009b30).stop 2023-11-15T05:01:33.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.961+0000 7f02a73f2700 1 -- 172.21.15.178:0/1221002274 shutdown_connections 2023-11-15T05:01:33.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.961+0000 7f02a73f2700 1 --2- 172.21.15.178:0/1221002274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f02a00f78e0 0x7f02a00f7d00 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:33.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.961+0000 7f02a73f2700 1 -- 172.21.15.178:0/1221002274 >> 172.21.15.178:0/1221002274 conn(0x7f02a00f0120 msgr2=0x7f02a00f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:33.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.961+0000 7f02a73f2700 1 -- 172.21.15.178:0/1221002274 shutdown_connections 2023-11-15T05:01:33.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.961+0000 7f02a73f2700 1 -- 172.21.15.178:0/1221002274 wait complete. 2023-11-15T05:01:33.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.961+0000 7f02a73f2700 1 Processor -- start 2023-11-15T05:01:33.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.961+0000 7f02a73f2700 1 -- start start 2023-11-15T05:01:33.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.961+0000 7f02a73f2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f02a00f78e0 0x7f02a01064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:33.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.961+0000 7f02a73f2700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a01069d0 con 0x7f02a00f78e0 2023-11-15T05:01:33.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.962+0000 7f02a63f0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f02a00f78e0 0x7f02a01064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:33.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.962+0000 7f02a63f0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f02a00f78e0 0x7f02a01064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:57008/0 (socket says 172.21.15.178:57008) 2023-11-15T05:01:33.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.962+0000 7f02a63f0700 1 -- 172.21.15.178:0/1817179937 learned_addr learned my addr 172.21.15.178:0/1817179937 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:33.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.962+0000 7f02a63f0700 1 -- 172.21.15.178:0/1817179937 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029c005040 con 0x7f02a00f78e0 2023-11-15T05:01:33.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.962+0000 7f02a63f0700 1 --2- 172.21.15.178:0/1817179937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f02a00f78e0 0x7f02a01064e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f029c013510 tx=0x7f029c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:33.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.962+0000 7f0296ffd700 1 -- 172.21.15.178:0/1817179937 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f029c009320 con 0x7f02a00f78e0 2023-11-15T05:01:33.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.962+0000 7f0296ffd700 1 -- 172.21.15.178:0/1817179937 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f029c013f00 con 0x7f02a00f78e0 2023-11-15T05:01:33.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.962+0000 7f0296ffd700 1 -- 172.21.15.178:0/1817179937 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f029c014dc0 con 0x7f02a00f78e0 2023-11-15T05:01:33.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.962+0000 7f02a73f2700 1 -- 172.21.15.178:0/1817179937 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02a0106bd0 con 0x7f02a00f78e0 2023-11-15T05:01:33.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.962+0000 7f02a73f2700 1 -- 172.21.15.178:0/1817179937 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02a0107060 con 0x7f02a00f78e0 2023-11-15T05:01:33.968 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.963+0000 7f0296ffd700 1 -- 172.21.15.178:0/1817179937 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f029c0095f0 con 0x7f02a00f78e0 2023-11-15T05:01:33.968 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.963+0000 7f0296ffd700 1 --2- 172.21.15.178:0/1817179937 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f028c02ffb0 0x7f028c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:33.969 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.963+0000 7f0296ffd700 1 -- 172.21.15.178:0/1817179937 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f029c0410b0 con 0x7f02a00f78e0 2023-11-15T05:01:33.969 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.963+0000 7f02a73f2700 1 -- 172.21.15.178:0/1817179937 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02a0045ad0 con 0x7f02a00f78e0 2023-11-15T05:01:33.969 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.963+0000 7f02a5bef700 1 --2- 172.21.15.178:0/1817179937 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f028c02ffb0 0x7f028c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:33.969 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.963+0000 7f02a5bef700 1 --2- 172.21.15.178:0/1817179937 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f028c02ffb0 0x7f028c032470 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f02900096a0 tx=0x7f0290006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:33.969 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:33.968+0000 7f0296ffd700 1 -- 172.21.15.178:0/1817179937 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f029c014420 con 0x7f02a00f78e0 2023-11-15T05:01:34.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:34.174+0000 7f02a73f2700 1 -- 172.21.15.178:0/1817179937 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02a0107d80 con 0x7f02a00f78e0 2023-11-15T05:01:34.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:34.175+0000 7f0296ffd700 1 -- 172.21.15.178:0/1817179937 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f029c02b020 con 0x7f02a00f78e0 2023-11-15T05:01:34.176 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:34.176 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:34.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:34.176+0000 7f02a73f2700 1 -- 172.21.15.178:0/1817179937 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f028c02ffb0 msgr2=0x7f028c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:34.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:34.176+0000 7f02a73f2700 1 --2- 172.21.15.178:0/1817179937 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f028c02ffb0 0x7f028c032470 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f02900096a0 tx=0x7f0290006b40).stop 2023-11-15T05:01:34.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:34.176+0000 7f02a73f2700 1 -- 172.21.15.178:0/1817179937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f02a00f78e0 msgr2=0x7f02a01064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:34.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:34.176+0000 7f02a73f2700 1 --2- 172.21.15.178:0/1817179937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f02a00f78e0 0x7f02a01064e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f029c013510 tx=0x7f029c00b040).stop 2023-11-15T05:01:34.178 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:34.177+0000 7f02a73f2700 1 -- 172.21.15.178:0/1817179937 shutdown_connections 2023-11-15T05:01:34.178 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:34.177+0000 7f02a73f2700 1 --2- 172.21.15.178:0/1817179937 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f028c02ffb0 0x7f028c032470 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:34.178 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:34.177+0000 7f02a73f2700 1 --2- 172.21.15.178:0/1817179937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f02a00f78e0 0x7f02a01064e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:34.178 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:34.177+0000 7f02a73f2700 1 -- 172.21.15.178:0/1817179937 >> 172.21.15.178:0/1817179937 conn(0x7f02a00f0120 msgr2=0x7f02a00f0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:34.178 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:34.177+0000 7f02a73f2700 1 -- 172.21.15.178:0/1817179937 shutdown_connections 2023-11-15T05:01:34.179 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:34.177+0000 7f02a73f2700 1 -- 172.21.15.178:0/1817179937 wait complete. 2023-11-15T05:01:34.179 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:01:34.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:33 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:32.366105+0000 mgr.smithi163.qesyrx (mgr.14162) 189 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:34.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:33 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:33.746242+0000 mon.smithi163 (mon.0) 257 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' 2023-11-15T05:01:35.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:34 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cephadm 2023-11-15T05:01:33.747648+0000 mgr.smithi163.qesyrx (mgr.14162) 190 : cephadm [INF] Deploying daemon node-exporter.smithi163 on smithi163 2023-11-15T05:01:35.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:34 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:34.175748+0000 mon.smithi163 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.178:0/1817179937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:01:35.575 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:35.575 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:36.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:34.366445+0000 mgr.smithi163.qesyrx (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:36.788 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.787+0000 7f3a2bfff700 1 -- 172.21.15.178:0/488541458 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a2c0f51a0 msgr2=0x7f3a2c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:36.789 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.787+0000 7f3a2bfff700 1 --2- 172.21.15.178:0/488541458 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a2c0f51a0 0x7f3a2c0f55c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f3a1c009fb0 tx=0x7f3a1c009b30).stop 2023-11-15T05:01:36.789 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.787+0000 7f3a2bfff700 1 -- 172.21.15.178:0/488541458 shutdown_connections 2023-11-15T05:01:36.789 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.787+0000 7f3a2bfff700 1 --2- 172.21.15.178:0/488541458 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a2c0f51a0 0x7f3a2c0f55c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:36.789 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.787+0000 7f3a2bfff700 1 -- 172.21.15.178:0/488541458 >> 172.21.15.178:0/488541458 conn(0x7f3a2c0f09a0 msgr2=0x7f3a2c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:36.789 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.787+0000 7f3a2bfff700 1 -- 172.21.15.178:0/488541458 shutdown_connections 2023-11-15T05:01:36.790 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.787+0000 7f3a2bfff700 1 -- 172.21.15.178:0/488541458 wait complete. 2023-11-15T05:01:36.790 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.787+0000 7f3a2bfff700 1 Processor -- start 2023-11-15T05:01:36.790 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.787+0000 7f3a2bfff700 1 -- start start 2023-11-15T05:01:36.790 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.788+0000 7f3a2bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a2c0f51a0 0x7f3a2c0fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:36.790 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.788+0000 7f3a2bfff700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2c0fb5a0 con 0x7f3a2c0f51a0 2023-11-15T05:01:36.791 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.788+0000 7f3a2affd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a2c0f51a0 0x7f3a2c0fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:36.791 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.788+0000 7f3a2affd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a2c0f51a0 0x7f3a2c0fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:52240/0 (socket says 172.21.15.178:52240) 2023-11-15T05:01:36.791 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.788+0000 7f3a2affd700 1 -- 172.21.15.178:0/4025326723 learned_addr learned my addr 172.21.15.178:0/4025326723 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:36.791 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.788+0000 7f3a2affd700 1 -- 172.21.15.178:0/4025326723 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1c005040 con 0x7f3a2c0f51a0 2023-11-15T05:01:36.791 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.788+0000 7f3a2affd700 1 --2- 172.21.15.178:0/4025326723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a2c0f51a0 0x7f3a2c0fd120 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f3a1c013510 tx=0x7f3a1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:36.793 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.788+0000 7f3a137fe700 1 -- 172.21.15.178:0/4025326723 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a1c009320 con 0x7f3a2c0f51a0 2023-11-15T05:01:36.793 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.789+0000 7f3a137fe700 1 -- 172.21.15.178:0/4025326723 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a1c013f00 con 0x7f3a2c0f51a0 2023-11-15T05:01:36.793 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.789+0000 7f3a2bfff700 1 -- 172.21.15.178:0/4025326723 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a2c0fb740 con 0x7f3a2c0f51a0 2023-11-15T05:01:36.793 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.789+0000 7f3a137fe700 1 -- 172.21.15.178:0/4025326723 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a1c014dc0 con 0x7f3a2c0f51a0 2023-11-15T05:01:36.793 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.789+0000 7f3a2bfff700 1 -- 172.21.15.178:0/4025326723 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a2c0fbbd0 con 0x7f3a2c0f51a0 2023-11-15T05:01:36.794 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.789+0000 7f3a137fe700 1 -- 172.21.15.178:0/4025326723 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3a1c014420 con 0x7f3a2c0f51a0 2023-11-15T05:01:36.794 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.789+0000 7f3a2bfff700 1 -- 172.21.15.178:0/4025326723 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a2c059100 con 0x7f3a2c0f51a0 2023-11-15T05:01:36.794 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.789+0000 7f3a137fe700 1 --2- 172.21.15.178:0/4025326723 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3a1402ff60 0x7f3a14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:36.796 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.790+0000 7f3a137fe700 1 -- 172.21.15.178:0/4025326723 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3a1c041110 con 0x7f3a2c0f51a0 2023-11-15T05:01:36.796 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.790+0000 7f3a2a7fc700 1 --2- 172.21.15.178:0/4025326723 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3a1402ff60 0x7f3a14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:36.796 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.790+0000 7f3a2a7fc700 1 --2- 172.21.15.178:0/4025326723 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3a1402ff60 0x7f3a14032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f3a200096a0 tx=0x7f3a20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:36.797 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.794+0000 7f3a137fe700 1 -- 172.21.15.178:0/4025326723 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a1c0095f0 con 0x7f3a2c0f51a0 2023-11-15T05:01:36.999 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:36.999+0000 7f3a2bfff700 1 -- 172.21.15.178:0/4025326723 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a2c0fabe0 con 0x7f3a2c0f51a0 2023-11-15T05:01:37.000 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:37.000+0000 7f3a137fe700 1 -- 172.21.15.178:0/4025326723 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3a1c01cb10 con 0x7f3a2c0f51a0 2023-11-15T05:01:37.000 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:37.000 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:37.001 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:37.001+0000 7f3a2bfff700 1 -- 172.21.15.178:0/4025326723 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3a1402ff60 msgr2=0x7f3a14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:37.002 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:37.001+0000 7f3a2bfff700 1 --2- 172.21.15.178:0/4025326723 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3a1402ff60 0x7f3a14032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f3a200096a0 tx=0x7f3a20006b40).stop 2023-11-15T05:01:37.002 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:37.001+0000 7f3a2bfff700 1 -- 172.21.15.178:0/4025326723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a2c0f51a0 msgr2=0x7f3a2c0fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:37.002 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:37.001+0000 7f3a2bfff700 1 --2- 172.21.15.178:0/4025326723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a2c0f51a0 0x7f3a2c0fd120 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f3a1c013510 tx=0x7f3a1c00b040).stop 2023-11-15T05:01:37.002 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:37.001+0000 7f3a2bfff700 1 -- 172.21.15.178:0/4025326723 shutdown_connections 2023-11-15T05:01:37.003 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:37.001+0000 7f3a2bfff700 1 --2- 172.21.15.178:0/4025326723 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3a1402ff60 0x7f3a14032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:37.003 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:37.001+0000 7f3a2bfff700 1 --2- 172.21.15.178:0/4025326723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a2c0f51a0 0x7f3a2c0fd120 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:37.003 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:37.001+0000 7f3a2bfff700 1 -- 172.21.15.178:0/4025326723 >> 172.21.15.178:0/4025326723 conn(0x7f3a2c0f09a0 msgr2=0x7f3a2c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:37.003 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:37.001+0000 7f3a2bfff700 1 -- 172.21.15.178:0/4025326723 shutdown_connections 2023-11-15T05:01:37.003 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:37.001+0000 7f3a2bfff700 1 -- 172.21.15.178:0/4025326723 wait complete. 2023-11-15T05:01:37.004 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:01:38.386 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:38.386 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:38.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:37 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:36.366788+0000 mgr.smithi163.qesyrx (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:38.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:37 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:37.000414+0000 mon.smithi163 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.178:0/4025326723' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:01:39.587 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.586+0000 7fb9a0896700 1 -- 172.21.15.178:0/2953159766 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb99c0f6a10 msgr2=0x7fb99c0f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:39.588 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.586+0000 7fb9a0896700 1 --2- 172.21.15.178:0/2953159766 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb99c0f6a10 0x7fb99c0f6e30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fb984009fb0 tx=0x7fb984009b30).stop 2023-11-15T05:01:39.588 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.586+0000 7fb9a0896700 1 -- 172.21.15.178:0/2953159766 shutdown_connections 2023-11-15T05:01:39.588 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.586+0000 7fb9a0896700 1 --2- 172.21.15.178:0/2953159766 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb99c0f6a10 0x7fb99c0f6e30 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:39.588 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.586+0000 7fb9a0896700 1 -- 172.21.15.178:0/2953159766 >> 172.21.15.178:0/2953159766 conn(0x7fb99c0f02e0 msgr2=0x7fb99c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:39.588 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.586+0000 7fb9a0896700 1 -- 172.21.15.178:0/2953159766 shutdown_connections 2023-11-15T05:01:39.589 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.586+0000 7fb9a0896700 1 -- 172.21.15.178:0/2953159766 wait complete. 2023-11-15T05:01:39.589 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.587+0000 7fb9a0896700 1 Processor -- start 2023-11-15T05:01:39.589 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.587+0000 7fb9a0896700 1 -- start start 2023-11-15T05:01:39.589 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.587+0000 7fb9a0896700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb99c0f6a10 0x7fb99c1055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:39.589 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.587+0000 7fb9a0896700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c105ac0 con 0x7fb99c0f6a10 2023-11-15T05:01:39.591 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.588+0000 7fb99a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb99c0f6a10 0x7fb99c1055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:39.591 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.588+0000 7fb99a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb99c0f6a10 0x7fb99c1055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:52246/0 (socket says 172.21.15.178:52246) 2023-11-15T05:01:39.591 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.588+0000 7fb99a59c700 1 -- 172.21.15.178:0/2036259037 learned_addr learned my addr 172.21.15.178:0/2036259037 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:39.592 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.588+0000 7fb99a59c700 1 -- 172.21.15.178:0/2036259037 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb984005040 con 0x7fb99c0f6a10 2023-11-15T05:01:39.592 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.588+0000 7fb99a59c700 1 --2- 172.21.15.178:0/2036259037 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb99c0f6a10 0x7fb99c1055d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fb984013510 tx=0x7fb98400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:39.592 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.588+0000 7fb992ffd700 1 -- 172.21.15.178:0/2036259037 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb984009320 con 0x7fb99c0f6a10 2023-11-15T05:01:39.592 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.588+0000 7fb992ffd700 1 -- 172.21.15.178:0/2036259037 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb984013f00 con 0x7fb99c0f6a10 2023-11-15T05:01:39.592 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.588+0000 7fb9a0896700 1 -- 172.21.15.178:0/2036259037 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb99c105cc0 con 0x7fb99c0f6a10 2023-11-15T05:01:39.593 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.588+0000 7fb992ffd700 1 -- 172.21.15.178:0/2036259037 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb984014d90 con 0x7fb99c0f6a10 2023-11-15T05:01:39.593 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.588+0000 7fb9a0896700 1 -- 172.21.15.178:0/2036259037 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb99c106090 con 0x7fb99c0f6a10 2023-11-15T05:01:39.593 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.589+0000 7fb992ffd700 1 -- 172.21.15.178:0/2036259037 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb9840263a0 con 0x7fb99c0f6a10 2023-11-15T05:01:39.593 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.589+0000 7fb9a0896700 1 -- 172.21.15.178:0/2036259037 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb99c059100 con 0x7fb99c0f6a10 2023-11-15T05:01:39.593 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.589+0000 7fb992ffd700 1 --2- 172.21.15.178:0/2036259037 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb98802ff60 0x7fb988032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:39.594 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.589+0000 7fb992ffd700 1 -- 172.21.15.178:0/2036259037 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb984041020 con 0x7fb99c0f6a10 2023-11-15T05:01:39.594 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.590+0000 7fb999d9b700 1 --2- 172.21.15.178:0/2036259037 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb98802ff60 0x7fb988032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:39.594 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.590+0000 7fb999d9b700 1 --2- 172.21.15.178:0/2036259037 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb98802ff60 0x7fb988032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fb98c0096a0 tx=0x7fb98c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:39.595 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.595+0000 7fb992ffd700 1 -- 172.21.15.178:0/2036259037 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9840126f0 con 0x7fb99c0f6a10 2023-11-15T05:01:39.797 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.797+0000 7fb9a0896700 1 -- 172.21.15.178:0/2036259037 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb99c045ad0 con 0x7fb99c0f6a10 2023-11-15T05:01:39.798 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.798+0000 7fb992ffd700 1 -- 172.21.15.178:0/2036259037 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb99c045ad0 con 0x7fb99c0f6a10 2023-11-15T05:01:39.799 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:39.799 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:39.800 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.799+0000 7fb9a0896700 1 -- 172.21.15.178:0/2036259037 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb98802ff60 msgr2=0x7fb988032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:39.800 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.799+0000 7fb9a0896700 1 --2- 172.21.15.178:0/2036259037 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb98802ff60 0x7fb988032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fb98c0096a0 tx=0x7fb98c006b40).stop 2023-11-15T05:01:39.800 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.799+0000 7fb9a0896700 1 -- 172.21.15.178:0/2036259037 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb99c0f6a10 msgr2=0x7fb99c1055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:39.800 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.799+0000 7fb9a0896700 1 --2- 172.21.15.178:0/2036259037 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb99c0f6a10 0x7fb99c1055d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fb984013510 tx=0x7fb98400b040).stop 2023-11-15T05:01:39.801 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.799+0000 7fb9a0896700 1 -- 172.21.15.178:0/2036259037 shutdown_connections 2023-11-15T05:01:39.801 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.799+0000 7fb9a0896700 1 --2- 172.21.15.178:0/2036259037 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb98802ff60 0x7fb988032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:39.801 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.799+0000 7fb9a0896700 1 --2- 172.21.15.178:0/2036259037 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb99c0f6a10 0x7fb99c1055d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:39.801 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.799+0000 7fb9a0896700 1 -- 172.21.15.178:0/2036259037 >> 172.21.15.178:0/2036259037 conn(0x7fb99c0f02e0 msgr2=0x7fb99c0f0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:39.801 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.799+0000 7fb9a0896700 1 -- 172.21.15.178:0/2036259037 shutdown_connections 2023-11-15T05:01:39.802 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:39.799+0000 7fb9a0896700 1 -- 172.21.15.178:0/2036259037 wait complete. 2023-11-15T05:01:39.802 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:01:40.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:39 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:38.367143+0000 mgr.smithi163.qesyrx (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:40.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:39 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:39.798483+0000 mon.smithi163 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.178:0/2036259037' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:01:41.228 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:41.228 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:42.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.433+0000 7fce1ae55700 1 -- 172.21.15.178:0/1715166481 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce140f6510 msgr2=0x7fce140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:42.436 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.433+0000 7fce1ae55700 1 --2- 172.21.15.178:0/1715166481 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce140f6510 0x7fce140f6930 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fce08009fb0 tx=0x7fce08009b30).stop 2023-11-15T05:01:42.436 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.433+0000 7fce1ae55700 1 -- 172.21.15.178:0/1715166481 shutdown_connections 2023-11-15T05:01:42.436 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.433+0000 7fce1ae55700 1 --2- 172.21.15.178:0/1715166481 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce140f6510 0x7fce140f6930 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:42.436 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.433+0000 7fce1ae55700 1 -- 172.21.15.178:0/1715166481 >> 172.21.15.178:0/1715166481 conn(0x7fce140f1c70 msgr2=0x7fce140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:42.436 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.433+0000 7fce1ae55700 1 -- 172.21.15.178:0/1715166481 shutdown_connections 2023-11-15T05:01:42.437 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.433+0000 7fce1ae55700 1 -- 172.21.15.178:0/1715166481 wait complete. 2023-11-15T05:01:42.437 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.434+0000 7fce1ae55700 1 Processor -- start 2023-11-15T05:01:42.437 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.434+0000 7fce1ae55700 1 -- start start 2023-11-15T05:01:42.437 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.434+0000 7fce1ae55700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce140f6510 0x7fce1406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:42.437 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.434+0000 7fce1ae55700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce14069e20 con 0x7fce140f6510 2023-11-15T05:01:42.438 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.434+0000 7fce18bf1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce140f6510 0x7fce1406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:42.438 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.434+0000 7fce18bf1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce140f6510 0x7fce1406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:52268/0 (socket says 172.21.15.178:52268) 2023-11-15T05:01:42.438 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.434+0000 7fce18bf1700 1 -- 172.21.15.178:0/3409599317 learned_addr learned my addr 172.21.15.178:0/3409599317 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:42.438 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.434+0000 7fce18bf1700 1 -- 172.21.15.178:0/3409599317 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce08005040 con 0x7fce140f6510 2023-11-15T05:01:42.439 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.434+0000 7fce18bf1700 1 --2- 172.21.15.178:0/3409599317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce140f6510 0x7fce1406b9a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fce08013510 tx=0x7fce0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:42.439 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.435+0000 7fce117fa700 1 -- 172.21.15.178:0/3409599317 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce08009320 con 0x7fce140f6510 2023-11-15T05:01:42.439 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.435+0000 7fce1ae55700 1 -- 172.21.15.178:0/3409599317 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce14069fc0 con 0x7fce140f6510 2023-11-15T05:01:42.439 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.435+0000 7fce117fa700 1 -- 172.21.15.178:0/3409599317 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce08013f00 con 0x7fce140f6510 2023-11-15T05:01:42.439 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.435+0000 7fce117fa700 1 -- 172.21.15.178:0/3409599317 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce08014d90 con 0x7fce140f6510 2023-11-15T05:01:42.440 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.435+0000 7fce1ae55700 1 -- 172.21.15.178:0/3409599317 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce1406a450 con 0x7fce140f6510 2023-11-15T05:01:42.441 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.435+0000 7fce117fa700 1 -- 172.21.15.178:0/3409599317 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce080263a0 con 0x7fce140f6510 2023-11-15T05:01:42.441 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.436+0000 7fce117fa700 1 --2- 172.21.15.178:0/3409599317 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fce0402ffb0 0x7fce04032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:42.441 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.436+0000 7fce1ae55700 1 -- 172.21.15.178:0/3409599317 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce14059100 con 0x7fce140f6510 2023-11-15T05:01:42.441 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.436+0000 7fce117fa700 1 -- 172.21.15.178:0/3409599317 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce08041070 con 0x7fce140f6510 2023-11-15T05:01:42.442 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.436+0000 7fce13fff700 1 --2- 172.21.15.178:0/3409599317 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fce0402ffb0 0x7fce04032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:42.442 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.436+0000 7fce13fff700 1 --2- 172.21.15.178:0/3409599317 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fce0402ffb0 0x7fce04032470 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fce000096a0 tx=0x7fce00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:42.442 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.441+0000 7fce117fa700 1 -- 172.21.15.178:0/3409599317 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce080095f0 con 0x7fce140f6510 2023-11-15T05:01:42.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:41 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:40.367604+0000 mgr.smithi163.qesyrx (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:42.644 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.644+0000 7fce1ae55700 1 -- 172.21.15.178:0/3409599317 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce140fbf60 con 0x7fce140f6510 2023-11-15T05:01:42.645 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.645+0000 7fce117fa700 1 -- 172.21.15.178:0/3409599317 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fce0801db70 con 0x7fce140f6510 2023-11-15T05:01:42.646 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:42.646 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:42.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.646+0000 7fce1ae55700 1 -- 172.21.15.178:0/3409599317 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fce0402ffb0 msgr2=0x7fce04032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:42.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.646+0000 7fce1ae55700 1 --2- 172.21.15.178:0/3409599317 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fce0402ffb0 0x7fce04032470 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fce000096a0 tx=0x7fce00006b40).stop 2023-11-15T05:01:42.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.646+0000 7fce1ae55700 1 -- 172.21.15.178:0/3409599317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce140f6510 msgr2=0x7fce1406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:42.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.646+0000 7fce1ae55700 1 --2- 172.21.15.178:0/3409599317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce140f6510 0x7fce1406b9a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fce08013510 tx=0x7fce0800b040).stop 2023-11-15T05:01:42.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.646+0000 7fce1ae55700 1 -- 172.21.15.178:0/3409599317 shutdown_connections 2023-11-15T05:01:42.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.646+0000 7fce1ae55700 1 --2- 172.21.15.178:0/3409599317 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fce0402ffb0 0x7fce04032470 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:42.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.646+0000 7fce1ae55700 1 --2- 172.21.15.178:0/3409599317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce140f6510 0x7fce1406b9a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:42.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.646+0000 7fce1ae55700 1 -- 172.21.15.178:0/3409599317 >> 172.21.15.178:0/3409599317 conn(0x7fce140f1c70 msgr2=0x7fce140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:42.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.646+0000 7fce1ae55700 1 -- 172.21.15.178:0/3409599317 shutdown_connections 2023-11-15T05:01:42.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:42.646+0000 7fce1ae55700 1 -- 172.21.15.178:0/3409599317 wait complete. 2023-11-15T05:01:42.649 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:01:43.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:42 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:42.645489+0000 mon.smithi163 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.178:0/3409599317' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:01:44.040 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:44.041 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:44.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:43 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:42.368016+0000 mgr.smithi163.qesyrx (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:45.217 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.217+0000 7faaee431700 1 -- 172.21.15.178:0/2085822017 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faae80f6a00 msgr2=0x7faae80f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:45.218 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.217+0000 7faaee431700 1 --2- 172.21.15.178:0/2085822017 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faae80f6a00 0x7faae80f6e20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7faadc009fb0 tx=0x7faadc009b30).stop 2023-11-15T05:01:45.218 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.217+0000 7faaee431700 1 -- 172.21.15.178:0/2085822017 shutdown_connections 2023-11-15T05:01:45.218 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.217+0000 7faaee431700 1 --2- 172.21.15.178:0/2085822017 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faae80f6a00 0x7faae80f6e20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:45.219 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.217+0000 7faaee431700 1 -- 172.21.15.178:0/2085822017 >> 172.21.15.178:0/2085822017 conn(0x7faae80f0260 msgr2=0x7faae80f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:45.219 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.217+0000 7faaee431700 1 -- 172.21.15.178:0/2085822017 shutdown_connections 2023-11-15T05:01:45.219 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.217+0000 7faaee431700 1 -- 172.21.15.178:0/2085822017 wait complete. 2023-11-15T05:01:45.221 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.218+0000 7faaee431700 1 Processor -- start 2023-11-15T05:01:45.221 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.218+0000 7faaee431700 1 -- start start 2023-11-15T05:01:45.222 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.218+0000 7faaee431700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faae80f6a00 0x7faae81093b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:45.222 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.218+0000 7faaee431700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae80f52b0 con 0x7faae80f6a00 2023-11-15T05:01:45.222 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.218+0000 7faae7fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faae80f6a00 0x7faae81093b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:45.222 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.218+0000 7faae7fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faae80f6a00 0x7faae81093b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54270/0 (socket says 172.21.15.178:54270) 2023-11-15T05:01:45.222 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.219+0000 7faae7fff700 1 -- 172.21.15.178:0/3619730653 learned_addr learned my addr 172.21.15.178:0/3619730653 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:45.223 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.219+0000 7faae7fff700 1 -- 172.21.15.178:0/3619730653 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faadc005040 con 0x7faae80f6a00 2023-11-15T05:01:45.223 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.219+0000 7faae7fff700 1 --2- 172.21.15.178:0/3619730653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faae80f6a00 0x7faae81093b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7faadc013510 tx=0x7faadc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:45.223 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.219+0000 7faae4ff9700 1 -- 172.21.15.178:0/3619730653 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faadc009320 con 0x7faae80f6a00 2023-11-15T05:01:45.223 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.219+0000 7faae4ff9700 1 -- 172.21.15.178:0/3619730653 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faadc013f00 con 0x7faae80f6a00 2023-11-15T05:01:45.224 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.219+0000 7faaee431700 1 -- 172.21.15.178:0/3619730653 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae80f54b0 con 0x7faae80f6a00 2023-11-15T05:01:45.224 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.219+0000 7faae4ff9700 1 -- 172.21.15.178:0/3619730653 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faadc014dc0 con 0x7faae80f6a00 2023-11-15T05:01:45.224 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.219+0000 7faaee431700 1 -- 172.21.15.178:0/3619730653 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae8109a60 con 0x7faae80f6a00 2023-11-15T05:01:45.224 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.220+0000 7faae4ff9700 1 -- 172.21.15.178:0/3619730653 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faadc014420 con 0x7faae80f6a00 2023-11-15T05:01:45.225 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.220+0000 7faaee431700 1 -- 172.21.15.178:0/3619730653 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae8045ad0 con 0x7faae80f6a00 2023-11-15T05:01:45.225 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.220+0000 7faae4ff9700 1 --2- 172.21.15.178:0/3619730653 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7faad002ff60 0x7faad0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:45.225 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.220+0000 7faae4ff9700 1 -- 172.21.15.178:0/3619730653 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faadc041030 con 0x7faae80f6a00 2023-11-15T05:01:45.225 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.220+0000 7faae77fe700 1 --2- 172.21.15.178:0/3619730653 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7faad002ff60 0x7faad0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:45.225 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.221+0000 7faae77fe700 1 --2- 172.21.15.178:0/3619730653 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7faad002ff60 0x7faad0032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7faad80096a0 tx=0x7faad8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:45.226 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.225+0000 7faae4ff9700 1 -- 172.21.15.178:0/3619730653 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faadc0095f0 con 0x7faae80f6a00 2023-11-15T05:01:45.432 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.432+0000 7faaee431700 1 -- 172.21.15.178:0/3619730653 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faae80faef0 con 0x7faae80f6a00 2023-11-15T05:01:45.433 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.433+0000 7faae4ff9700 1 -- 172.21.15.178:0/3619730653 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7faadc01cb10 con 0x7faae80f6a00 2023-11-15T05:01:45.434 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:45.434 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:45.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.435+0000 7faaee431700 1 -- 172.21.15.178:0/3619730653 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7faad002ff60 msgr2=0x7faad0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:45.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.435+0000 7faaee431700 1 --2- 172.21.15.178:0/3619730653 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7faad002ff60 0x7faad0032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7faad80096a0 tx=0x7faad8006b40).stop 2023-11-15T05:01:45.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.435+0000 7faaee431700 1 -- 172.21.15.178:0/3619730653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faae80f6a00 msgr2=0x7faae81093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:45.436 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.435+0000 7faaee431700 1 --2- 172.21.15.178:0/3619730653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faae80f6a00 0x7faae81093b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7faadc013510 tx=0x7faadc00b040).stop 2023-11-15T05:01:45.436 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.435+0000 7faaee431700 1 -- 172.21.15.178:0/3619730653 shutdown_connections 2023-11-15T05:01:45.436 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.435+0000 7faaee431700 1 --2- 172.21.15.178:0/3619730653 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7faad002ff60 0x7faad0032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:45.436 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.435+0000 7faaee431700 1 --2- 172.21.15.178:0/3619730653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faae80f6a00 0x7faae81093b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:45.436 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.435+0000 7faaee431700 1 -- 172.21.15.178:0/3619730653 >> 172.21.15.178:0/3619730653 conn(0x7faae80f0260 msgr2=0x7faae80f0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:45.437 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.435+0000 7faaee431700 1 -- 172.21.15.178:0/3619730653 shutdown_connections 2023-11-15T05:01:45.437 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:45.435+0000 7faaee431700 1 -- 172.21.15.178:0/3619730653 wait complete. 2023-11-15T05:01:45.437 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:01:46.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:45 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:44.368325+0000 mgr.smithi163.qesyrx (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:46.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:45 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:45.433552+0000 mon.smithi163 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.178:0/3619730653' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:01:46.820 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:46.820 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:47.959 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.957+0000 7feb8a11d700 1 -- 172.21.15.178:0/2512413283 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb84060280 msgr2=0x7feb840f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:47.959 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.957+0000 7feb8a11d700 1 --2- 172.21.15.178:0/2512413283 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb84060280 0x7feb840f90f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7feb74009fb0 tx=0x7feb74009b30).stop 2023-11-15T05:01:47.959 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.957+0000 7feb8a11d700 1 -- 172.21.15.178:0/2512413283 shutdown_connections 2023-11-15T05:01:47.960 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.957+0000 7feb8a11d700 1 --2- 172.21.15.178:0/2512413283 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb84060280 0x7feb840f90f0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:47.960 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.957+0000 7feb8a11d700 1 -- 172.21.15.178:0/2512413283 >> 172.21.15.178:0/2512413283 conn(0x7feb840f0b70 msgr2=0x7feb840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:47.960 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.957+0000 7feb8a11d700 1 -- 172.21.15.178:0/2512413283 shutdown_connections 2023-11-15T05:01:47.960 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.957+0000 7feb8a11d700 1 -- 172.21.15.178:0/2512413283 wait complete. 2023-11-15T05:01:47.960 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.958+0000 7feb8a11d700 1 Processor -- start 2023-11-15T05:01:47.961 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.958+0000 7feb8a11d700 1 -- start start 2023-11-15T05:01:47.961 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.958+0000 7feb8a11d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb84060280 0x7feb841066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:47.961 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.958+0000 7feb8a11d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb84106bd0 con 0x7feb84060280 2023-11-15T05:01:47.961 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.958+0000 7feb837fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb84060280 0x7feb841066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:47.962 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.958+0000 7feb837fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb84060280 0x7feb841066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54288/0 (socket says 172.21.15.178:54288) 2023-11-15T05:01:47.962 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.958+0000 7feb837fe700 1 -- 172.21.15.178:0/4030048677 learned_addr learned my addr 172.21.15.178:0/4030048677 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:47.962 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.958+0000 7feb837fe700 1 -- 172.21.15.178:0/4030048677 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb74005040 con 0x7feb84060280 2023-11-15T05:01:47.962 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.959+0000 7feb837fe700 1 --2- 172.21.15.178:0/4030048677 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb84060280 0x7feb841066e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7feb74009830 tx=0x7feb7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:47.962 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.959+0000 7feb7bfff700 1 -- 172.21.15.178:0/4030048677 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb74009320 con 0x7feb84060280 2023-11-15T05:01:47.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.959+0000 7feb8a11d700 1 -- 172.21.15.178:0/4030048677 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb84106dd0 con 0x7feb84060280 2023-11-15T05:01:47.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.959+0000 7feb7bfff700 1 -- 172.21.15.178:0/4030048677 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb740136c0 con 0x7feb84060280 2023-11-15T05:01:47.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.959+0000 7feb7bfff700 1 -- 172.21.15.178:0/4030048677 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb7401ee90 con 0x7feb84060280 2023-11-15T05:01:47.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.959+0000 7feb8a11d700 1 -- 172.21.15.178:0/4030048677 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb84107260 con 0x7feb84060280 2023-11-15T05:01:47.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.959+0000 7feb7bfff700 1 -- 172.21.15.178:0/4030048677 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb7401e420 con 0x7feb84060280 2023-11-15T05:01:47.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.960+0000 7feb8a11d700 1 -- 172.21.15.178:0/4030048677 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb84045ad0 con 0x7feb84060280 2023-11-15T05:01:47.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.960+0000 7feb7bfff700 1 --2- 172.21.15.178:0/4030048677 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feb7002ff10 0x7feb700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:47.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.960+0000 7feb7bfff700 1 -- 172.21.15.178:0/4030048677 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb74040f10 con 0x7feb84060280 2023-11-15T05:01:47.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.960+0000 7feb82ffd700 1 --2- 172.21.15.178:0/4030048677 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feb7002ff10 0x7feb700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:47.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.960+0000 7feb82ffd700 1 --2- 172.21.15.178:0/4030048677 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feb7002ff10 0x7feb700323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7feb6c0096a0 tx=0x7feb6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:47.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:47.965+0000 7feb7bfff700 1 -- 172.21.15.178:0/4030048677 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb740095f0 con 0x7feb84060280 2023-11-15T05:01:48.171 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:48.170+0000 7feb8a11d700 1 -- 172.21.15.178:0/4030048677 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb84108110 con 0x7feb84060280 2023-11-15T05:01:48.171 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:48.171+0000 7feb7bfff700 1 -- 172.21.15.178:0/4030048677 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7feb7402c2d0 con 0x7feb84060280 2023-11-15T05:01:48.172 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:48.172 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:48.174 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:48.173+0000 7feb8a11d700 1 -- 172.21.15.178:0/4030048677 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feb7002ff10 msgr2=0x7feb700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:48.174 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:48.173+0000 7feb8a11d700 1 --2- 172.21.15.178:0/4030048677 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feb7002ff10 0x7feb700323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7feb6c0096a0 tx=0x7feb6c006b40).stop 2023-11-15T05:01:48.174 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:48.173+0000 7feb8a11d700 1 -- 172.21.15.178:0/4030048677 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb84060280 msgr2=0x7feb841066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:48.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:48.173+0000 7feb8a11d700 1 --2- 172.21.15.178:0/4030048677 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb84060280 0x7feb841066e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7feb74009830 tx=0x7feb7400b040).stop 2023-11-15T05:01:48.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:48.173+0000 7feb8a11d700 1 -- 172.21.15.178:0/4030048677 shutdown_connections 2023-11-15T05:01:48.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:48.173+0000 7feb8a11d700 1 --2- 172.21.15.178:0/4030048677 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feb7002ff10 0x7feb700323d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:48.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:48.173+0000 7feb8a11d700 1 --2- 172.21.15.178:0/4030048677 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb84060280 0x7feb841066e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:48.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:48.173+0000 7feb8a11d700 1 -- 172.21.15.178:0/4030048677 >> 172.21.15.178:0/4030048677 conn(0x7feb840f0b70 msgr2=0x7feb840f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:48.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:48.173+0000 7feb8a11d700 1 -- 172.21.15.178:0/4030048677 shutdown_connections 2023-11-15T05:01:48.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:48.173+0000 7feb8a11d700 1 -- 172.21.15.178:0/4030048677 wait complete. 2023-11-15T05:01:48.177 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:01:48.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:47 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:46.368607+0000 mgr.smithi163.qesyrx (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:49.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:48 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:48.171592+0000 mon.smithi163 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.178:0/4030048677' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:01:49.589 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:49.590 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:50.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:49 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:48.368869+0000 mgr.smithi163.qesyrx (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:50.797 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.796+0000 7fa98054e700 1 -- 172.21.15.178:0/4030450428 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9780f8700 msgr2=0x7fa9780f8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:50.797 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.796+0000 7fa98054e700 1 --2- 172.21.15.178:0/4030450428 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9780f8700 0x7fa9780f8b20 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fa974009fb0 tx=0x7fa974009b30).stop 2023-11-15T05:01:50.797 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.796+0000 7fa98054e700 1 -- 172.21.15.178:0/4030450428 shutdown_connections 2023-11-15T05:01:50.797 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.796+0000 7fa98054e700 1 --2- 172.21.15.178:0/4030450428 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9780f8700 0x7fa9780f8b20 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:50.798 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.796+0000 7fa98054e700 1 -- 172.21.15.178:0/4030450428 >> 172.21.15.178:0/4030450428 conn(0x7fa9780f40e0 msgr2=0x7fa9780f64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:50.798 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.796+0000 7fa98054e700 1 -- 172.21.15.178:0/4030450428 shutdown_connections 2023-11-15T05:01:50.798 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.796+0000 7fa98054e700 1 -- 172.21.15.178:0/4030450428 wait complete. 2023-11-15T05:01:50.798 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.796+0000 7fa98054e700 1 Processor -- start 2023-11-15T05:01:50.799 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.796+0000 7fa98054e700 1 -- start start 2023-11-15T05:01:50.799 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.797+0000 7fa98054e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9780f8700 0x7fa978108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:50.799 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.797+0000 7fa98054e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa978108e10 con 0x7fa9780f8700 2023-11-15T05:01:50.801 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.797+0000 7fa97e2ea700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9780f8700 0x7fa978108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:50.801 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.797+0000 7fa97e2ea700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9780f8700 0x7fa978108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54310/0 (socket says 172.21.15.178:54310) 2023-11-15T05:01:50.801 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.797+0000 7fa97e2ea700 1 -- 172.21.15.178:0/3844247306 learned_addr learned my addr 172.21.15.178:0/3844247306 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:50.801 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.797+0000 7fa97e2ea700 1 -- 172.21.15.178:0/3844247306 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa974005040 con 0x7fa9780f8700 2023-11-15T05:01:50.802 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.797+0000 7fa97e2ea700 1 --2- 172.21.15.178:0/3844247306 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9780f8700 0x7fa978108920 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fa974014000 tx=0x7fa97400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:50.802 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.797+0000 7fa96effd700 1 -- 172.21.15.178:0/3844247306 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9740095f0 con 0x7fa9780f8700 2023-11-15T05:01:50.802 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.797+0000 7fa98054e700 1 -- 172.21.15.178:0/3844247306 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa978109010 con 0x7fa9780f8700 2023-11-15T05:01:50.802 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.797+0000 7fa96effd700 1 -- 172.21.15.178:0/3844247306 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa974007e70 con 0x7fa9780f8700 2023-11-15T05:01:50.802 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.797+0000 7fa96effd700 1 -- 172.21.15.178:0/3844247306 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa974014ea0 con 0x7fa9780f8700 2023-11-15T05:01:50.803 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.797+0000 7fa98054e700 1 -- 172.21.15.178:0/3844247306 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9781094a0 con 0x7fa9780f8700 2023-11-15T05:01:50.803 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.798+0000 7fa96effd700 1 -- 172.21.15.178:0/3844247306 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa974025070 con 0x7fa9780f8700 2023-11-15T05:01:50.803 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.798+0000 7fa98054e700 1 -- 172.21.15.178:0/3844247306 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa978045ad0 con 0x7fa9780f8700 2023-11-15T05:01:50.803 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.798+0000 7fa96effd700 1 --2- 172.21.15.178:0/3844247306 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa96402ff60 0x7fa964032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:50.803 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.798+0000 7fa96effd700 1 -- 172.21.15.178:0/3844247306 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa974040930 con 0x7fa9780f8700 2023-11-15T05:01:50.804 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.799+0000 7fa97dae9700 1 --2- 172.21.15.178:0/3844247306 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa96402ff60 0x7fa964032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:50.804 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.799+0000 7fa97dae9700 1 --2- 172.21.15.178:0/3844247306 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa96402ff60 0x7fa964032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fa9680096a0 tx=0x7fa968006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:50.804 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:50.803+0000 7fa96effd700 1 -- 172.21.15.178:0/3844247306 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa97400f4e0 con 0x7fa9780f8700 2023-11-15T05:01:51.009 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:51.009+0000 7fa98054e700 1 -- 172.21.15.178:0/3844247306 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9780fe150 con 0x7fa9780f8700 2023-11-15T05:01:51.010 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:51.009+0000 7fa96effd700 1 -- 172.21.15.178:0/3844247306 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa97400f300 con 0x7fa9780f8700 2023-11-15T05:01:51.010 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:51.010 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:51.011 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:51.011+0000 7fa98054e700 1 -- 172.21.15.178:0/3844247306 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa96402ff60 msgr2=0x7fa964032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:51.011 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:51.011+0000 7fa98054e700 1 --2- 172.21.15.178:0/3844247306 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa96402ff60 0x7fa964032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fa9680096a0 tx=0x7fa968006b40).stop 2023-11-15T05:01:51.011 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:51.011+0000 7fa98054e700 1 -- 172.21.15.178:0/3844247306 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9780f8700 msgr2=0x7fa978108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:51.012 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:51.011+0000 7fa98054e700 1 --2- 172.21.15.178:0/3844247306 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9780f8700 0x7fa978108920 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fa974014000 tx=0x7fa97400b040).stop 2023-11-15T05:01:51.012 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:51.011+0000 7fa98054e700 1 -- 172.21.15.178:0/3844247306 shutdown_connections 2023-11-15T05:01:51.012 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:51.011+0000 7fa98054e700 1 --2- 172.21.15.178:0/3844247306 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa96402ff60 0x7fa964032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:51.012 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:51.011+0000 7fa98054e700 1 --2- 172.21.15.178:0/3844247306 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9780f8700 0x7fa978108920 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:51.012 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:51.011+0000 7fa98054e700 1 -- 172.21.15.178:0/3844247306 >> 172.21.15.178:0/3844247306 conn(0x7fa9780f40e0 msgr2=0x7fa9780f5be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:51.013 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:51.011+0000 7fa98054e700 1 -- 172.21.15.178:0/3844247306 shutdown_connections 2023-11-15T05:01:51.013 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:51.011+0000 7fa98054e700 1 -- 172.21.15.178:0/3844247306 wait complete. 2023-11-15T05:01:51.013 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:01:52.402 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:52.403 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:52.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:51 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:50.369154+0000 mgr.smithi163.qesyrx (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:52.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:51 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:51.009952+0000 mon.smithi163 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.178:0/3844247306' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:01:53.678 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.677+0000 7f09ddb4d700 1 -- 172.21.15.178:0/1972795505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09d80f6510 msgr2=0x7f09d80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:53.678 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.677+0000 7f09ddb4d700 1 --2- 172.21.15.178:0/1972795505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09d80f6510 0x7f09d80f6930 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f09cc009fb0 tx=0x7f09cc009b30).stop 2023-11-15T05:01:53.678 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.677+0000 7f09ddb4d700 1 -- 172.21.15.178:0/1972795505 shutdown_connections 2023-11-15T05:01:53.679 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.677+0000 7f09ddb4d700 1 --2- 172.21.15.178:0/1972795505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09d80f6510 0x7f09d80f6930 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:53.679 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.677+0000 7f09ddb4d700 1 -- 172.21.15.178:0/1972795505 >> 172.21.15.178:0/1972795505 conn(0x7f09d80f1c70 msgr2=0x7f09d80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:53.679 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.677+0000 7f09ddb4d700 1 -- 172.21.15.178:0/1972795505 shutdown_connections 2023-11-15T05:01:53.679 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.677+0000 7f09ddb4d700 1 -- 172.21.15.178:0/1972795505 wait complete. 2023-11-15T05:01:53.679 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.677+0000 7f09ddb4d700 1 Processor -- start 2023-11-15T05:01:53.680 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.678+0000 7f09ddb4d700 1 -- start start 2023-11-15T05:01:53.680 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.678+0000 7f09ddb4d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09d80f6510 0x7f09d8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:53.680 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.678+0000 7f09ddb4d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d8106c20 con 0x7f09d80f6510 2023-11-15T05:01:53.682 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.678+0000 7f09d77fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09d80f6510 0x7f09d8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:53.682 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.678+0000 7f09d77fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09d80f6510 0x7f09d8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54332/0 (socket says 172.21.15.178:54332) 2023-11-15T05:01:53.682 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.678+0000 7f09d77fe700 1 -- 172.21.15.178:0/3584639071 learned_addr learned my addr 172.21.15.178:0/3584639071 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:53.682 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.679+0000 7f09d77fe700 1 -- 172.21.15.178:0/3584639071 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09cc005040 con 0x7f09d80f6510 2023-11-15T05:01:53.683 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.679+0000 7f09d77fe700 1 --2- 172.21.15.178:0/3584639071 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09d80f6510 0x7f09d8106730 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f09cc013510 tx=0x7f09cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:53.683 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.679+0000 7f09bffff700 1 -- 172.21.15.178:0/3584639071 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09cc009320 con 0x7f09d80f6510 2023-11-15T05:01:53.684 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.679+0000 7f09ddb4d700 1 -- 172.21.15.178:0/3584639071 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d8106e20 con 0x7f09d80f6510 2023-11-15T05:01:53.684 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.679+0000 7f09bffff700 1 -- 172.21.15.178:0/3584639071 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09cc013f00 con 0x7f09d80f6510 2023-11-15T05:01:53.684 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.679+0000 7f09bffff700 1 -- 172.21.15.178:0/3584639071 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09cc014d90 con 0x7f09d80f6510 2023-11-15T05:01:53.684 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.679+0000 7f09ddb4d700 1 -- 172.21.15.178:0/3584639071 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d81072b0 con 0x7f09d80f6510 2023-11-15T05:01:53.684 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.680+0000 7f09bffff700 1 -- 172.21.15.178:0/3584639071 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09cc014420 con 0x7f09d80f6510 2023-11-15T05:01:53.685 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.680+0000 7f09ddb4d700 1 -- 172.21.15.178:0/3584639071 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d8045ad0 con 0x7f09d80f6510 2023-11-15T05:01:53.685 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.680+0000 7f09bffff700 1 --2- 172.21.15.178:0/3584639071 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f09c002ff60 0x7f09c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:53.685 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.680+0000 7f09bffff700 1 -- 172.21.15.178:0/3584639071 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f09cc041000 con 0x7f09d80f6510 2023-11-15T05:01:53.686 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.680+0000 7f09d6ffd700 1 --2- 172.21.15.178:0/3584639071 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f09c002ff60 0x7f09c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:53.686 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.680+0000 7f09d6ffd700 1 --2- 172.21.15.178:0/3584639071 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f09c002ff60 0x7f09c0032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f09c80096a0 tx=0x7f09c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:53.687 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.685+0000 7f09bffff700 1 -- 172.21.15.178:0/3584639071 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09cc0095f0 con 0x7f09d80f6510 2023-11-15T05:01:53.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.891+0000 7f09ddb4d700 1 -- 172.21.15.178:0/3584639071 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09d80fbf60 con 0x7f09d80f6510 2023-11-15T05:01:53.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.892+0000 7f09bffff700 1 -- 172.21.15.178:0/3584639071 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f09cc01db70 con 0x7f09d80f6510 2023-11-15T05:01:53.893 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:53.893 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:53.895 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.894+0000 7f09ddb4d700 1 -- 172.21.15.178:0/3584639071 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f09c002ff60 msgr2=0x7f09c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:53.895 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.894+0000 7f09ddb4d700 1 --2- 172.21.15.178:0/3584639071 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f09c002ff60 0x7f09c0032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f09c80096a0 tx=0x7f09c8006b40).stop 2023-11-15T05:01:53.895 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.894+0000 7f09ddb4d700 1 -- 172.21.15.178:0/3584639071 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09d80f6510 msgr2=0x7f09d8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:53.895 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.894+0000 7f09ddb4d700 1 --2- 172.21.15.178:0/3584639071 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09d80f6510 0x7f09d8106730 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f09cc013510 tx=0x7f09cc00b040).stop 2023-11-15T05:01:53.896 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.894+0000 7f09ddb4d700 1 -- 172.21.15.178:0/3584639071 shutdown_connections 2023-11-15T05:01:53.896 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.894+0000 7f09ddb4d700 1 --2- 172.21.15.178:0/3584639071 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f09c002ff60 0x7f09c0032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:53.896 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.894+0000 7f09ddb4d700 1 --2- 172.21.15.178:0/3584639071 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09d80f6510 0x7f09d8106730 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:53.896 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.894+0000 7f09ddb4d700 1 -- 172.21.15.178:0/3584639071 >> 172.21.15.178:0/3584639071 conn(0x7f09d80f1c70 msgr2=0x7f09d80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:53.896 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.894+0000 7f09ddb4d700 1 -- 172.21.15.178:0/3584639071 shutdown_connections 2023-11-15T05:01:53.897 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:53.894+0000 7f09ddb4d700 1 -- 172.21.15.178:0/3584639071 wait complete. 2023-11-15T05:01:53.897 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:01:54.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:53 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:52.369512+0000 mgr.smithi163.qesyrx (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:54.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:53 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:53.892664+0000 mon.smithi163 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.178:0/3584639071' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:01:55.286 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:55.286 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:56.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:55 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:54.369810+0000 mgr.smithi163.qesyrx (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:56.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.489+0000 7f5d2f2c2700 1 -- 172.21.15.178:0/3909685309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d28060280 msgr2=0x7f5d280f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:56.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.489+0000 7f5d2f2c2700 1 --2- 172.21.15.178:0/3909685309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d28060280 0x7f5d280f90f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f5d24009960 tx=0x7f5d24004d10).stop 2023-11-15T05:01:56.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.489+0000 7f5d2f2c2700 1 -- 172.21.15.178:0/3909685309 shutdown_connections 2023-11-15T05:01:56.492 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.489+0000 7f5d2f2c2700 1 --2- 172.21.15.178:0/3909685309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d28060280 0x7f5d280f90f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:56.492 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.489+0000 7f5d2f2c2700 1 -- 172.21.15.178:0/3909685309 >> 172.21.15.178:0/3909685309 conn(0x7f5d280f0b70 msgr2=0x7f5d280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:56.492 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.489+0000 7f5d2f2c2700 1 -- 172.21.15.178:0/3909685309 shutdown_connections 2023-11-15T05:01:56.492 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.490+0000 7f5d2f2c2700 1 -- 172.21.15.178:0/3909685309 wait complete. 2023-11-15T05:01:56.493 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.490+0000 7f5d2f2c2700 1 Processor -- start 2023-11-15T05:01:56.493 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.490+0000 7f5d2f2c2700 1 -- start start 2023-11-15T05:01:56.493 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.490+0000 7f5d2f2c2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d28060280 0x7f5d281066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:56.493 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.490+0000 7f5d2f2c2700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d28106bd0 con 0x7f5d28060280 2023-11-15T05:01:56.493 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.490+0000 7f5d2d05e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d28060280 0x7f5d281066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:56.494 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.490+0000 7f5d2d05e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d28060280 0x7f5d281066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:47472/0 (socket says 172.21.15.178:47472) 2023-11-15T05:01:56.494 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.490+0000 7f5d2d05e700 1 -- 172.21.15.178:0/690023177 learned_addr learned my addr 172.21.15.178:0/690023177 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:56.494 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.491+0000 7f5d2d05e700 1 -- 172.21.15.178:0/690023177 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d24009640 con 0x7f5d28060280 2023-11-15T05:01:56.494 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.491+0000 7f5d2d05e700 1 --2- 172.21.15.178:0/690023177 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d28060280 0x7f5d281066e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f5d24007bd0 tx=0x7f5d24004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:56.496 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.491+0000 7f5d1dffb700 1 -- 172.21.15.178:0/690023177 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d24018070 con 0x7f5d28060280 2023-11-15T05:01:56.496 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.491+0000 7f5d1dffb700 1 -- 172.21.15.178:0/690023177 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d2400b870 con 0x7f5d28060280 2023-11-15T05:01:56.496 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.491+0000 7f5d2f2c2700 1 -- 172.21.15.178:0/690023177 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d28106dd0 con 0x7f5d28060280 2023-11-15T05:01:56.497 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.491+0000 7f5d1dffb700 1 -- 172.21.15.178:0/690023177 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d2401d420 con 0x7f5d28060280 2023-11-15T05:01:56.497 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.491+0000 7f5d2f2c2700 1 -- 172.21.15.178:0/690023177 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d28107260 con 0x7f5d28060280 2023-11-15T05:01:56.497 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.492+0000 7f5d1dffb700 1 -- 172.21.15.178:0/690023177 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d24024060 con 0x7f5d28060280 2023-11-15T05:01:56.497 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.492+0000 7f5d2f2c2700 1 -- 172.21.15.178:0/690023177 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d28045ad0 con 0x7f5d28060280 2023-11-15T05:01:56.497 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.492+0000 7f5d1dffb700 1 --2- 172.21.15.178:0/690023177 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5d14030000 0x7f5d140324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:56.498 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.492+0000 7f5d1dffb700 1 -- 172.21.15.178:0/690023177 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d240410a0 con 0x7f5d28060280 2023-11-15T05:01:56.498 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.493+0000 7f5d2c85d700 1 --2- 172.21.15.178:0/690023177 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5d14030000 0x7f5d140324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:56.498 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.493+0000 7f5d2c85d700 1 --2- 172.21.15.178:0/690023177 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5d14030000 0x7f5d140324c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f5d180096a0 tx=0x7f5d18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:56.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.497+0000 7f5d1dffb700 1 -- 172.21.15.178:0/690023177 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d24011970 con 0x7f5d28060280 2023-11-15T05:01:56.701 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.701+0000 7f5d2f2c2700 1 -- 172.21.15.178:0/690023177 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d28108070 con 0x7f5d28060280 2023-11-15T05:01:56.702 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.702+0000 7f5d1dffb700 1 -- 172.21.15.178:0/690023177 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5d2402c2d0 con 0x7f5d28060280 2023-11-15T05:01:56.702 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:56.703 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:56.704 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.703+0000 7f5d2f2c2700 1 -- 172.21.15.178:0/690023177 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5d14030000 msgr2=0x7f5d140324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:56.705 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.703+0000 7f5d2f2c2700 1 --2- 172.21.15.178:0/690023177 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5d14030000 0x7f5d140324c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f5d180096a0 tx=0x7f5d18006b40).stop 2023-11-15T05:01:56.705 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.703+0000 7f5d2f2c2700 1 -- 172.21.15.178:0/690023177 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d28060280 msgr2=0x7f5d281066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:56.705 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.703+0000 7f5d2f2c2700 1 --2- 172.21.15.178:0/690023177 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d28060280 0x7f5d281066e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f5d24007bd0 tx=0x7f5d24004d10).stop 2023-11-15T05:01:56.705 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.703+0000 7f5d2f2c2700 1 -- 172.21.15.178:0/690023177 shutdown_connections 2023-11-15T05:01:56.705 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.703+0000 7f5d2f2c2700 1 --2- 172.21.15.178:0/690023177 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5d14030000 0x7f5d140324c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:56.706 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.703+0000 7f5d2f2c2700 1 --2- 172.21.15.178:0/690023177 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d28060280 0x7f5d281066e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:56.706 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.703+0000 7f5d2f2c2700 1 -- 172.21.15.178:0/690023177 >> 172.21.15.178:0/690023177 conn(0x7f5d280f0b70 msgr2=0x7f5d280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:56.706 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.704+0000 7f5d2f2c2700 1 -- 172.21.15.178:0/690023177 shutdown_connections 2023-11-15T05:01:56.706 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:56.704+0000 7f5d2f2c2700 1 -- 172.21.15.178:0/690023177 wait complete. 2023-11-15T05:01:56.707 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:01:57.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:56 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:56.702618+0000 mon.smithi163 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.178:0/690023177' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:01:58.097 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:58.097 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:58.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:57 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:56.370161+0000 mgr.smithi163.qesyrx (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:59.353 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.351+0000 7f8945b94700 1 -- 172.21.15.178:0/3497026055 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8940060280 msgr2=0x7f89400f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:59.353 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.351+0000 7f8945b94700 1 --2- 172.21.15.178:0/3497026055 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8940060280 0x7f89400f90f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f8930009fb0 tx=0x7f8930009b30).stop 2023-11-15T05:01:59.354 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.351+0000 7f8945b94700 1 -- 172.21.15.178:0/3497026055 shutdown_connections 2023-11-15T05:01:59.354 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.351+0000 7f8945b94700 1 --2- 172.21.15.178:0/3497026055 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8940060280 0x7f89400f90f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:59.354 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.351+0000 7f8945b94700 1 -- 172.21.15.178:0/3497026055 >> 172.21.15.178:0/3497026055 conn(0x7f89400f0b70 msgr2=0x7f89400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:59.354 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.351+0000 7f8945b94700 1 -- 172.21.15.178:0/3497026055 shutdown_connections 2023-11-15T05:01:59.355 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.351+0000 7f8945b94700 1 -- 172.21.15.178:0/3497026055 wait complete. 2023-11-15T05:01:59.355 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.352+0000 7f8945b94700 1 Processor -- start 2023-11-15T05:01:59.355 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.352+0000 7f8945b94700 1 -- start start 2023-11-15T05:01:59.355 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.352+0000 7f8945b94700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8940060280 0x7f89400f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:59.355 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.352+0000 7f8945b94700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89400f4ff0 con 0x7f8940060280 2023-11-15T05:01:59.356 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.352+0000 7f893f7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8940060280 0x7f89400f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:59.356 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.352+0000 7f893f7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8940060280 0x7f89400f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:47494/0 (socket says 172.21.15.178:47494) 2023-11-15T05:01:59.356 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.352+0000 7f893f7fe700 1 -- 172.21.15.178:0/3604811493 learned_addr learned my addr 172.21.15.178:0/3604811493 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:01:59.356 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.352+0000 7f893f7fe700 1 -- 172.21.15.178:0/3604811493 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8930005040 con 0x7f8940060280 2023-11-15T05:01:59.356 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.353+0000 7f893f7fe700 1 --2- 172.21.15.178:0/3604811493 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8940060280 0x7f89400f6b70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f8930009830 tx=0x7f893000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:59.357 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.353+0000 7f8937fff700 1 -- 172.21.15.178:0/3604811493 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8930009320 con 0x7f8940060280 2023-11-15T05:01:59.357 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.353+0000 7f8937fff700 1 -- 172.21.15.178:0/3604811493 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f89300136c0 con 0x7f8940060280 2023-11-15T05:01:59.357 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.353+0000 7f8945b94700 1 -- 172.21.15.178:0/3604811493 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89400f5190 con 0x7f8940060280 2023-11-15T05:01:59.357 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.353+0000 7f8937fff700 1 -- 172.21.15.178:0/3604811493 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f893001ee90 con 0x7f8940060280 2023-11-15T05:01:59.358 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.353+0000 7f8945b94700 1 -- 172.21.15.178:0/3604811493 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89400f5620 con 0x7f8940060280 2023-11-15T05:01:59.359 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.353+0000 7f8937fff700 1 -- 172.21.15.178:0/3604811493 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8930014440 con 0x7f8940060280 2023-11-15T05:01:59.359 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.354+0000 7f8945b94700 1 -- 172.21.15.178:0/3604811493 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8940059100 con 0x7f8940060280 2023-11-15T05:01:59.359 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.354+0000 7f8937fff700 1 --2- 172.21.15.178:0/3604811493 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f892c02ff10 0x7f892c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:59.360 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.354+0000 7f8937fff700 1 -- 172.21.15.178:0/3604811493 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8930041310 con 0x7f8940060280 2023-11-15T05:01:59.360 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.354+0000 7f893effd700 1 --2- 172.21.15.178:0/3604811493 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f892c02ff10 0x7f892c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:59.360 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.354+0000 7f893effd700 1 --2- 172.21.15.178:0/3604811493 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f892c02ff10 0x7f892c0323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f89280096a0 tx=0x7f8928006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:59.360 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.359+0000 7f8937fff700 1 -- 172.21.15.178:0/3604811493 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f893001c910 con 0x7f8940060280 2023-11-15T05:01:59.566 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.565+0000 7f8945b94700 1 -- 172.21.15.178:0/3604811493 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f89400fbf50 con 0x7f8940060280 2023-11-15T05:01:59.566 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.566+0000 7f8937fff700 1 -- 172.21.15.178:0/3604811493 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f893001c320 con 0x7f8940060280 2023-11-15T05:01:59.567 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:01:59.567 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:01:59.569 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.568+0000 7f8945b94700 1 -- 172.21.15.178:0/3604811493 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f892c02ff10 msgr2=0x7f892c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:59.569 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.568+0000 7f8945b94700 1 --2- 172.21.15.178:0/3604811493 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f892c02ff10 0x7f892c0323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f89280096a0 tx=0x7f8928006b40).stop 2023-11-15T05:01:59.570 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.568+0000 7f8945b94700 1 -- 172.21.15.178:0/3604811493 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8940060280 msgr2=0x7f89400f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:59.570 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.568+0000 7f8945b94700 1 --2- 172.21.15.178:0/3604811493 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8940060280 0x7f89400f6b70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f8930009830 tx=0x7f893000b040).stop 2023-11-15T05:01:59.570 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.568+0000 7f8945b94700 1 -- 172.21.15.178:0/3604811493 shutdown_connections 2023-11-15T05:01:59.571 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.568+0000 7f8945b94700 1 --2- 172.21.15.178:0/3604811493 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f892c02ff10 0x7f892c0323d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:59.571 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.568+0000 7f8945b94700 1 --2- 172.21.15.178:0/3604811493 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8940060280 0x7f89400f6b70 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:59.571 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.568+0000 7f8945b94700 1 -- 172.21.15.178:0/3604811493 >> 172.21.15.178:0/3604811493 conn(0x7f89400f0b70 msgr2=0x7f89400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:59.572 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.568+0000 7f8945b94700 1 -- 172.21.15.178:0/3604811493 shutdown_connections 2023-11-15T05:01:59.572 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:01:59.568+0000 7f8945b94700 1 -- 172.21.15.178:0/3604811493 wait complete. 2023-11-15T05:01:59.572 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:00.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:59 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:01:58.370513+0000 mgr.smithi163.qesyrx (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:00.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:01:59 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:01:59.566811+0000 mon.smithi163 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.178:0/3604811493' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:00.952 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:00.952 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:02.118 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.115+0000 7f93ce6b3700 1 -- 172.21.15.178:0/2035788511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f93c80f5410 msgr2=0x7f93c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:02.118 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.115+0000 7f93ce6b3700 1 --2- 172.21.15.178:0/2035788511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f93c80f5410 0x7f93c80f5830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f93b8009fb0 tx=0x7f93b8009b30).stop 2023-11-15T05:02:02.118 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.115+0000 7f93ce6b3700 1 -- 172.21.15.178:0/2035788511 shutdown_connections 2023-11-15T05:02:02.118 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.115+0000 7f93ce6b3700 1 --2- 172.21.15.178:0/2035788511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f93c80f5410 0x7f93c80f5830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:02.119 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.115+0000 7f93ce6b3700 1 -- 172.21.15.178:0/2035788511 >> 172.21.15.178:0/2035788511 conn(0x7f93c80f0b70 msgr2=0x7f93c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:02.119 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.115+0000 7f93ce6b3700 1 -- 172.21.15.178:0/2035788511 shutdown_connections 2023-11-15T05:02:02.119 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.115+0000 7f93ce6b3700 1 -- 172.21.15.178:0/2035788511 wait complete. 2023-11-15T05:02:02.119 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.115+0000 7f93ce6b3700 1 Processor -- start 2023-11-15T05:02:02.119 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.115+0000 7f93ce6b3700 1 -- start start 2023-11-15T05:02:02.120 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.115+0000 7f93ce6b3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f93c80f5410 0x7f93c80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:02.120 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.115+0000 7f93ce6b3700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c80fb820 con 0x7f93c80f5410 2023-11-15T05:02:02.120 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.116+0000 7f93c7fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f93c80f5410 0x7f93c80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:02.120 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.116+0000 7f93c7fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f93c80f5410 0x7f93c80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:47514/0 (socket says 172.21.15.178:47514) 2023-11-15T05:02:02.120 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.116+0000 7f93c7fff700 1 -- 172.21.15.178:0/2666036564 learned_addr learned my addr 172.21.15.178:0/2666036564 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:02:02.121 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.116+0000 7f93c7fff700 1 -- 172.21.15.178:0/2666036564 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b8005040 con 0x7f93c80f5410 2023-11-15T05:02:02.121 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.116+0000 7f93c7fff700 1 --2- 172.21.15.178:0/2666036564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f93c80f5410 0x7f93c80fd3a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f93b8013510 tx=0x7f93b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:02.121 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.116+0000 7f93c4ff9700 1 -- 172.21.15.178:0/2666036564 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93b8009320 con 0x7f93c80f5410 2023-11-15T05:02:02.121 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.116+0000 7f93ce6b3700 1 -- 172.21.15.178:0/2666036564 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c80fb9c0 con 0x7f93c80f5410 2023-11-15T05:02:02.121 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.116+0000 7f93c4ff9700 1 -- 172.21.15.178:0/2666036564 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93b8013f00 con 0x7f93c80f5410 2023-11-15T05:02:02.122 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.116+0000 7f93c4ff9700 1 -- 172.21.15.178:0/2666036564 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93b8014d90 con 0x7f93c80f5410 2023-11-15T05:02:02.122 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.117+0000 7f93ce6b3700 1 -- 172.21.15.178:0/2666036564 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93c80fbe50 con 0x7f93c80f5410 2023-11-15T05:02:02.123 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.117+0000 7f93c4ff9700 1 -- 172.21.15.178:0/2666036564 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93b80095f0 con 0x7f93c80f5410 2023-11-15T05:02:02.123 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.117+0000 7f93c4ff9700 1 --2- 172.21.15.178:0/2666036564 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f93b002ff60 0x7f93b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:02.123 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.117+0000 7f93c4ff9700 1 -- 172.21.15.178:0/2666036564 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f93b8041000 con 0x7f93c80f5410 2023-11-15T05:02:02.124 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.117+0000 7f93ce6b3700 1 -- 172.21.15.178:0/2666036564 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93c8059100 con 0x7f93c80f5410 2023-11-15T05:02:02.124 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.117+0000 7f93c77fe700 1 --2- 172.21.15.178:0/2666036564 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f93b002ff60 0x7f93b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:02.124 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.118+0000 7f93c77fe700 1 --2- 172.21.15.178:0/2666036564 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f93b002ff60 0x7f93b0032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f93bc0096a0 tx=0x7f93bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:02.125 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.122+0000 7f93c4ff9700 1 -- 172.21.15.178:0/2666036564 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f93b8014420 con 0x7f93c80f5410 2023-11-15T05:02:02.330 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.329+0000 7f93ce6b3700 1 -- 172.21.15.178:0/2666036564 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93c80fc4b0 con 0x7f93c80f5410 2023-11-15T05:02:02.331 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.329+0000 7f93c4ff9700 1 -- 172.21.15.178:0/2666036564 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f93b802b020 con 0x7f93c80f5410 2023-11-15T05:02:02.331 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:02:02.332 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:02.332 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.331+0000 7f93ce6b3700 1 -- 172.21.15.178:0/2666036564 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f93b002ff60 msgr2=0x7f93b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:02.333 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.331+0000 7f93ce6b3700 1 --2- 172.21.15.178:0/2666036564 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f93b002ff60 0x7f93b0032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f93bc0096a0 tx=0x7f93bc006b40).stop 2023-11-15T05:02:02.333 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.331+0000 7f93ce6b3700 1 -- 172.21.15.178:0/2666036564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f93c80f5410 msgr2=0x7f93c80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:02.333 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.331+0000 7f93ce6b3700 1 --2- 172.21.15.178:0/2666036564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f93c80f5410 0x7f93c80fd3a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f93b8013510 tx=0x7f93b800b040).stop 2023-11-15T05:02:02.333 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.331+0000 7f93ce6b3700 1 -- 172.21.15.178:0/2666036564 shutdown_connections 2023-11-15T05:02:02.333 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.331+0000 7f93ce6b3700 1 --2- 172.21.15.178:0/2666036564 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f93b002ff60 0x7f93b0032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:02.334 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.331+0000 7f93ce6b3700 1 --2- 172.21.15.178:0/2666036564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f93c80f5410 0x7f93c80fd3a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:02.334 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.331+0000 7f93ce6b3700 1 -- 172.21.15.178:0/2666036564 >> 172.21.15.178:0/2666036564 conn(0x7f93c80f0b70 msgr2=0x7f93c805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:02.334 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.331+0000 7f93ce6b3700 1 -- 172.21.15.178:0/2666036564 shutdown_connections 2023-11-15T05:02:02.334 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:02.331+0000 7f93ce6b3700 1 -- 172.21.15.178:0/2666036564 wait complete. 2023-11-15T05:02:02.335 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:02.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:01 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:00.370870+0000 mgr.smithi163.qesyrx (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:03.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:03 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:02.331216+0000 mon.smithi163 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.178:0/2666036564' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:03.747 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:03.748 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:04.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:04 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:02.371227+0000 mgr.smithi163.qesyrx (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:04.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.960+0000 7f05c0a78700 1 -- 172.21.15.178:0/2896266807 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05bc0f47c0 msgr2=0x7f05bc0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:04.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.960+0000 7f05c0a78700 1 --2- 172.21.15.178:0/2896266807 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05bc0f47c0 0x7f05bc0f4be0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f05b0009fb0 tx=0x7f05b0009b30).stop 2023-11-15T05:02:04.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.960+0000 7f05c0a78700 1 -- 172.21.15.178:0/2896266807 shutdown_connections 2023-11-15T05:02:04.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.960+0000 7f05c0a78700 1 --2- 172.21.15.178:0/2896266807 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05bc0f47c0 0x7f05bc0f4be0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:04.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.960+0000 7f05c0a78700 1 -- 172.21.15.178:0/2896266807 >> 172.21.15.178:0/2896266807 conn(0x7f05bc0f02e0 msgr2=0x7f05bc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:04.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.960+0000 7f05c0a78700 1 -- 172.21.15.178:0/2896266807 shutdown_connections 2023-11-15T05:02:04.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.960+0000 7f05c0a78700 1 -- 172.21.15.178:0/2896266807 wait complete. 2023-11-15T05:02:04.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.961+0000 7f05c0a78700 1 Processor -- start 2023-11-15T05:02:04.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.961+0000 7f05c0a78700 1 -- start start 2023-11-15T05:02:04.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.961+0000 7f05c0a78700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05bc0f47c0 0x7f05bc106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:04.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.961+0000 7f05c0a78700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc106bf0 con 0x7f05bc0f47c0 2023-11-15T05:02:04.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.962+0000 7f05ba59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05bc0f47c0 0x7f05bc106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:04.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.962+0000 7f05ba59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05bc0f47c0 0x7f05bc106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:58466/0 (socket says 172.21.15.178:58466) 2023-11-15T05:02:04.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.962+0000 7f05ba59c700 1 -- 172.21.15.178:0/2813121753 learned_addr learned my addr 172.21.15.178:0/2813121753 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:02:04.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.962+0000 7f05ba59c700 1 -- 172.21.15.178:0/2813121753 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b0005040 con 0x7f05bc0f47c0 2023-11-15T05:02:04.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.962+0000 7f05ba59c700 1 --2- 172.21.15.178:0/2813121753 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05bc0f47c0 0x7f05bc106700 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f05b0014000 tx=0x7f05b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:04.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.962+0000 7f05aaffd700 1 -- 172.21.15.178:0/2813121753 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05b00095f0 con 0x7f05bc0f47c0 2023-11-15T05:02:04.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.962+0000 7f05c0a78700 1 -- 172.21.15.178:0/2813121753 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05bc106df0 con 0x7f05bc0f47c0 2023-11-15T05:02:04.968 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.962+0000 7f05aaffd700 1 -- 172.21.15.178:0/2813121753 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f05b0007e70 con 0x7f05bc0f47c0 2023-11-15T05:02:04.968 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.962+0000 7f05aaffd700 1 -- 172.21.15.178:0/2813121753 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05b0014ea0 con 0x7f05bc0f47c0 2023-11-15T05:02:04.968 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.962+0000 7f05c0a78700 1 -- 172.21.15.178:0/2813121753 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05bc107280 con 0x7f05bc0f47c0 2023-11-15T05:02:04.968 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.963+0000 7f05aaffd700 1 -- 172.21.15.178:0/2813121753 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f05b0014690 con 0x7f05bc0f47c0 2023-11-15T05:02:04.968 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.963+0000 7f05c0a78700 1 -- 172.21.15.178:0/2813121753 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05bc045ad0 con 0x7f05bc0f47c0 2023-11-15T05:02:04.969 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.963+0000 7f05aaffd700 1 --2- 172.21.15.178:0/2813121753 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f05a402ff60 0x7f05a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:04.969 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.963+0000 7f05aaffd700 1 -- 172.21.15.178:0/2813121753 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f05b0040870 con 0x7f05bc0f47c0 2023-11-15T05:02:04.969 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.963+0000 7f05b9d9b700 1 --2- 172.21.15.178:0/2813121753 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f05a402ff60 0x7f05a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:04.969 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.964+0000 7f05b9d9b700 1 --2- 172.21.15.178:0/2813121753 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f05a402ff60 0x7f05a4032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f05ac0096a0 tx=0x7f05ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:04.970 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:04.968+0000 7f05aaffd700 1 -- 172.21.15.178:0/2813121753 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05b000f300 con 0x7f05bc0f47c0 2023-11-15T05:02:05.172 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:05.171+0000 7f05c0a78700 1 -- 172.21.15.178:0/2813121753 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f05bc108130 con 0x7f05bc0f47c0 2023-11-15T05:02:05.173 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:05.172+0000 7f05aaffd700 1 -- 172.21.15.178:0/2813121753 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f05b000f300 con 0x7f05bc0f47c0 2023-11-15T05:02:05.173 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:02:05.173 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:05.174 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:05.173+0000 7f05c0a78700 1 -- 172.21.15.178:0/2813121753 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f05a402ff60 msgr2=0x7f05a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:05.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:05.173+0000 7f05c0a78700 1 --2- 172.21.15.178:0/2813121753 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f05a402ff60 0x7f05a4032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f05ac0096a0 tx=0x7f05ac006b40).stop 2023-11-15T05:02:05.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:05.173+0000 7f05c0a78700 1 -- 172.21.15.178:0/2813121753 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05bc0f47c0 msgr2=0x7f05bc106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:05.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:05.173+0000 7f05c0a78700 1 --2- 172.21.15.178:0/2813121753 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05bc0f47c0 0x7f05bc106700 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f05b0014000 tx=0x7f05b000b040).stop 2023-11-15T05:02:05.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:05.173+0000 7f05c0a78700 1 -- 172.21.15.178:0/2813121753 shutdown_connections 2023-11-15T05:02:05.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:05.173+0000 7f05c0a78700 1 --2- 172.21.15.178:0/2813121753 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f05a402ff60 0x7f05a4032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:05.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:05.173+0000 7f05c0a78700 1 --2- 172.21.15.178:0/2813121753 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05bc0f47c0 0x7f05bc106700 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:05.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:05.173+0000 7f05c0a78700 1 -- 172.21.15.178:0/2813121753 >> 172.21.15.178:0/2813121753 conn(0x7f05bc0f02e0 msgr2=0x7f05bc0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:05.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:05.173+0000 7f05c0a78700 1 -- 172.21.15.178:0/2813121753 shutdown_connections 2023-11-15T05:02:05.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:05.173+0000 7f05c0a78700 1 -- 172.21.15.178:0/2813121753 wait complete. 2023-11-15T05:02:05.177 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:06.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:06 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:04.371468+0000 mgr.smithi163.qesyrx (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:06.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:06 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:05.173374+0000 mon.smithi163 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.178:0/2813121753' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:06.582 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:06.583 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:07.845 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.843+0000 7fe252104700 1 -- 172.21.15.178:0/2884248150 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe24c060280 msgr2=0x7fe24c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:07.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.843+0000 7fe252104700 1 --2- 172.21.15.178:0/2884248150 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe24c060280 0x7fe24c0f90f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fe234009fb0 tx=0x7fe234009b30).stop 2023-11-15T05:02:07.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.843+0000 7fe252104700 1 -- 172.21.15.178:0/2884248150 shutdown_connections 2023-11-15T05:02:07.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.843+0000 7fe252104700 1 --2- 172.21.15.178:0/2884248150 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe24c060280 0x7fe24c0f90f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:07.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.843+0000 7fe252104700 1 -- 172.21.15.178:0/2884248150 >> 172.21.15.178:0/2884248150 conn(0x7fe24c0f0b70 msgr2=0x7fe24c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:07.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.843+0000 7fe252104700 1 -- 172.21.15.178:0/2884248150 shutdown_connections 2023-11-15T05:02:07.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.843+0000 7fe252104700 1 -- 172.21.15.178:0/2884248150 wait complete. 2023-11-15T05:02:07.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.843+0000 7fe252104700 1 Processor -- start 2023-11-15T05:02:07.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.844+0000 7fe252104700 1 -- start start 2023-11-15T05:02:07.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.844+0000 7fe252104700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe24c060280 0x7fe24c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:07.848 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.844+0000 7fe252104700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c106bd0 con 0x7fe24c060280 2023-11-15T05:02:07.848 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.844+0000 7fe24b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe24c060280 0x7fe24c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:07.848 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.844+0000 7fe24b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe24c060280 0x7fe24c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:58476/0 (socket says 172.21.15.178:58476) 2023-11-15T05:02:07.848 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.844+0000 7fe24b7fe700 1 -- 172.21.15.178:0/723913935 learned_addr learned my addr 172.21.15.178:0/723913935 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:02:07.848 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.844+0000 7fe24b7fe700 1 -- 172.21.15.178:0/723913935 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe234005040 con 0x7fe24c060280 2023-11-15T05:02:07.849 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.844+0000 7fe24b7fe700 1 --2- 172.21.15.178:0/723913935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe24c060280 0x7fe24c1066e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fe234013510 tx=0x7fe23400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:07.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.845+0000 7fe243fff700 1 -- 172.21.15.178:0/723913935 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe234009320 con 0x7fe24c060280 2023-11-15T05:02:07.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.845+0000 7fe243fff700 1 -- 172.21.15.178:0/723913935 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe234013f00 con 0x7fe24c060280 2023-11-15T05:02:07.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.845+0000 7fe252104700 1 -- 172.21.15.178:0/723913935 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe24c106dd0 con 0x7fe24c060280 2023-11-15T05:02:07.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.845+0000 7fe243fff700 1 -- 172.21.15.178:0/723913935 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe234014dc0 con 0x7fe24c060280 2023-11-15T05:02:07.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.845+0000 7fe252104700 1 -- 172.21.15.178:0/723913935 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe24c107260 con 0x7fe24c060280 2023-11-15T05:02:07.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.845+0000 7fe243fff700 1 -- 172.21.15.178:0/723913935 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe2340095f0 con 0x7fe24c060280 2023-11-15T05:02:07.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.846+0000 7fe252104700 1 -- 172.21.15.178:0/723913935 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe24c045ad0 con 0x7fe24c060280 2023-11-15T05:02:07.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.846+0000 7fe243fff700 1 --2- 172.21.15.178:0/723913935 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe23802ff60 0x7fe238032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:07.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.846+0000 7fe243fff700 1 -- 172.21.15.178:0/723913935 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe234041080 con 0x7fe24c060280 2023-11-15T05:02:07.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.846+0000 7fe24affd700 1 --2- 172.21.15.178:0/723913935 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe23802ff60 0x7fe238032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:07.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.846+0000 7fe24affd700 1 --2- 172.21.15.178:0/723913935 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe23802ff60 0x7fe238032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fe23c0096a0 tx=0x7fe23c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:07.853 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:07.851+0000 7fe243fff700 1 -- 172.21.15.178:0/723913935 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe234014420 con 0x7fe24c060280 2023-11-15T05:02:08.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:08.055+0000 7fe252104700 1 -- 172.21.15.178:0/723913935 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe24c108110 con 0x7fe24c060280 2023-11-15T05:02:08.057 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:08.056+0000 7fe243fff700 1 -- 172.21.15.178:0/723913935 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe234010230 con 0x7fe24c060280 2023-11-15T05:02:08.057 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:02:08.058 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:08.062 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:08.057+0000 7fe252104700 1 -- 172.21.15.178:0/723913935 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe23802ff60 msgr2=0x7fe238032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:08.062 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:08.057+0000 7fe252104700 1 --2- 172.21.15.178:0/723913935 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe23802ff60 0x7fe238032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fe23c0096a0 tx=0x7fe23c006b40).stop 2023-11-15T05:02:08.062 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:08.057+0000 7fe252104700 1 -- 172.21.15.178:0/723913935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe24c060280 msgr2=0x7fe24c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:08.063 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:08.057+0000 7fe252104700 1 --2- 172.21.15.178:0/723913935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe24c060280 0x7fe24c1066e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fe234013510 tx=0x7fe23400b040).stop 2023-11-15T05:02:08.063 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:08.058+0000 7fe252104700 1 -- 172.21.15.178:0/723913935 shutdown_connections 2023-11-15T05:02:08.063 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:08.058+0000 7fe252104700 1 --2- 172.21.15.178:0/723913935 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe23802ff60 0x7fe238032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:08.063 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:08.058+0000 7fe252104700 1 --2- 172.21.15.178:0/723913935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe24c060280 0x7fe24c1066e0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:08.063 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:08.058+0000 7fe252104700 1 -- 172.21.15.178:0/723913935 >> 172.21.15.178:0/723913935 conn(0x7fe24c0f0b70 msgr2=0x7fe24c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:08.064 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:08.058+0000 7fe252104700 1 -- 172.21.15.178:0/723913935 shutdown_connections 2023-11-15T05:02:08.064 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:08.058+0000 7fe252104700 1 -- 172.21.15.178:0/723913935 wait complete. 2023-11-15T05:02:08.064 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:08.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:08 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:06.371776+0000 mgr.smithi163.qesyrx (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:09.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:09 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:08.057609+0000 mon.smithi163 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.178:0/723913935' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:09.463 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:09.463 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:10.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:10 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:08.372101+0000 mgr.smithi163.qesyrx (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:10.770 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.769+0000 7f12fd584700 1 -- 172.21.15.178:0/396192606 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12f80f5410 msgr2=0x7f12f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:10.771 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.769+0000 7f12fd584700 1 --2- 172.21.15.178:0/396192606 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12f80f5410 0x7f12f80f5830 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f12ec009fb0 tx=0x7f12ec009b30).stop 2023-11-15T05:02:10.771 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.769+0000 7f12fd584700 1 -- 172.21.15.178:0/396192606 shutdown_connections 2023-11-15T05:02:10.771 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.769+0000 7f12fd584700 1 --2- 172.21.15.178:0/396192606 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12f80f5410 0x7f12f80f5830 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:10.771 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.769+0000 7f12fd584700 1 -- 172.21.15.178:0/396192606 >> 172.21.15.178:0/396192606 conn(0x7f12f80f0b70 msgr2=0x7f12f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:10.772 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.769+0000 7f12fd584700 1 -- 172.21.15.178:0/396192606 shutdown_connections 2023-11-15T05:02:10.772 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.769+0000 7f12fd584700 1 -- 172.21.15.178:0/396192606 wait complete. 2023-11-15T05:02:10.774 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.770+0000 7f12fd584700 1 Processor -- start 2023-11-15T05:02:10.774 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.770+0000 7f12fd584700 1 -- start start 2023-11-15T05:02:10.774 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.770+0000 7f12fd584700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12f80f5410 0x7f12f81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:10.774 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.770+0000 7f12fd584700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f8106bd0 con 0x7f12f80f5410 2023-11-15T05:02:10.775 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.770+0000 7f12f6ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12f80f5410 0x7f12f81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:10.775 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.770+0000 7f12f6ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12f80f5410 0x7f12f81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:58500/0 (socket says 172.21.15.178:58500) 2023-11-15T05:02:10.775 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.770+0000 7f12f6ffd700 1 -- 172.21.15.178:0/3700128710 learned_addr learned my addr 172.21.15.178:0/3700128710 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:02:10.775 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.770+0000 7f12f6ffd700 1 -- 172.21.15.178:0/3700128710 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ec005040 con 0x7f12f80f5410 2023-11-15T05:02:10.775 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.770+0000 7f12f6ffd700 1 --2- 172.21.15.178:0/3700128710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12f80f5410 0x7f12f81066e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f12ec013510 tx=0x7f12ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:10.776 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.771+0000 7f12df7fe700 1 -- 172.21.15.178:0/3700128710 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12ec009320 con 0x7f12f80f5410 2023-11-15T05:02:10.776 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.771+0000 7f12fd584700 1 -- 172.21.15.178:0/3700128710 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f8106dd0 con 0x7f12f80f5410 2023-11-15T05:02:10.776 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.771+0000 7f12df7fe700 1 -- 172.21.15.178:0/3700128710 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12ec013f00 con 0x7f12f80f5410 2023-11-15T05:02:10.776 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.771+0000 7f12df7fe700 1 -- 172.21.15.178:0/3700128710 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12ec014d90 con 0x7f12f80f5410 2023-11-15T05:02:10.776 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.771+0000 7f12fd584700 1 -- 172.21.15.178:0/3700128710 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f8107260 con 0x7f12f80f5410 2023-11-15T05:02:10.777 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.771+0000 7f12df7fe700 1 -- 172.21.15.178:0/3700128710 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12ec014420 con 0x7f12f80f5410 2023-11-15T05:02:10.777 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.771+0000 7f12fd584700 1 -- 172.21.15.178:0/3700128710 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f8045ad0 con 0x7f12f80f5410 2023-11-15T05:02:10.777 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.772+0000 7f12df7fe700 1 --2- 172.21.15.178:0/3700128710 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f12e002ff60 0x7f12e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:10.777 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.772+0000 7f12df7fe700 1 -- 172.21.15.178:0/3700128710 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12ec040ff0 con 0x7f12f80f5410 2023-11-15T05:02:10.777 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.772+0000 7f12f67fc700 1 --2- 172.21.15.178:0/3700128710 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f12e002ff60 0x7f12e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:10.777 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.772+0000 7f12f67fc700 1 --2- 172.21.15.178:0/3700128710 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f12e002ff60 0x7f12e0032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f12e80096a0 tx=0x7f12e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:10.778 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.776+0000 7f12df7fe700 1 -- 172.21.15.178:0/3700128710 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12ec0095f0 con 0x7f12f80f5410 2023-11-15T05:02:10.983 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.982+0000 7f12fd584700 1 -- 172.21.15.178:0/3700128710 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12f80fae60 con 0x7f12f80f5410 2023-11-15T05:02:10.984 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.983+0000 7f12df7fe700 1 -- 172.21.15.178:0/3700128710 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f12ec01db70 con 0x7f12f80f5410 2023-11-15T05:02:10.984 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:02:10.984 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:10.985 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.984+0000 7f12fd584700 1 -- 172.21.15.178:0/3700128710 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f12e002ff60 msgr2=0x7f12e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:10.986 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.984+0000 7f12fd584700 1 --2- 172.21.15.178:0/3700128710 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f12e002ff60 0x7f12e0032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f12e80096a0 tx=0x7f12e8006b40).stop 2023-11-15T05:02:10.986 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.984+0000 7f12fd584700 1 -- 172.21.15.178:0/3700128710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12f80f5410 msgr2=0x7f12f81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:10.986 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.984+0000 7f12fd584700 1 --2- 172.21.15.178:0/3700128710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12f80f5410 0x7f12f81066e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f12ec013510 tx=0x7f12ec00b040).stop 2023-11-15T05:02:10.986 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.984+0000 7f12fd584700 1 -- 172.21.15.178:0/3700128710 shutdown_connections 2023-11-15T05:02:10.987 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.984+0000 7f12fd584700 1 --2- 172.21.15.178:0/3700128710 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f12e002ff60 0x7f12e0032420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:10.987 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.984+0000 7f12fd584700 1 --2- 172.21.15.178:0/3700128710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12f80f5410 0x7f12f81066e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:10.988 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.984+0000 7f12fd584700 1 -- 172.21.15.178:0/3700128710 >> 172.21.15.178:0/3700128710 conn(0x7f12f80f0b70 msgr2=0x7f12f805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:10.988 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.984+0000 7f12fd584700 1 -- 172.21.15.178:0/3700128710 shutdown_connections 2023-11-15T05:02:10.988 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:10.984+0000 7f12fd584700 1 -- 172.21.15.178:0/3700128710 wait complete. 2023-11-15T05:02:10.988 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:11.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:11 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:10.984234+0000 mon.smithi163 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.178:0/3700128710' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:12.366 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:12.367 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:12.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:12 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:10.372332+0000 mgr.smithi163.qesyrx (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:13.578 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.575+0000 7f028e6e3700 1 -- 172.21.15.178:0/589511531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0288060280 msgr2=0x7f02880f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:13.578 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.575+0000 7f028e6e3700 1 --2- 172.21.15.178:0/589511531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0288060280 0x7f02880f90f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f027c009fb0 tx=0x7f027c009b30).stop 2023-11-15T05:02:13.579 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.575+0000 7f028e6e3700 1 -- 172.21.15.178:0/589511531 shutdown_connections 2023-11-15T05:02:13.579 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.575+0000 7f028e6e3700 1 --2- 172.21.15.178:0/589511531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0288060280 0x7f02880f90f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:13.579 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.575+0000 7f028e6e3700 1 -- 172.21.15.178:0/589511531 >> 172.21.15.178:0/589511531 conn(0x7f02880f0b70 msgr2=0x7f02880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:13.579 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.575+0000 7f028e6e3700 1 -- 172.21.15.178:0/589511531 shutdown_connections 2023-11-15T05:02:13.579 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.575+0000 7f028e6e3700 1 -- 172.21.15.178:0/589511531 wait complete. 2023-11-15T05:02:13.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.576+0000 7f028e6e3700 1 Processor -- start 2023-11-15T05:02:13.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.576+0000 7f028e6e3700 1 -- start start 2023-11-15T05:02:13.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.576+0000 7f028e6e3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0288060280 0x7f02881066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:13.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.576+0000 7f028e6e3700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0288106bd0 con 0x7f0288060280 2023-11-15T05:02:13.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.576+0000 7f0287fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0288060280 0x7f02881066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:13.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.576+0000 7f0287fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0288060280 0x7f02881066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:58516/0 (socket says 172.21.15.178:58516) 2023-11-15T05:02:13.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.576+0000 7f0287fff700 1 -- 172.21.15.178:0/1563269069 learned_addr learned my addr 172.21.15.178:0/1563269069 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:02:13.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.576+0000 7f0287fff700 1 -- 172.21.15.178:0/1563269069 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027c005040 con 0x7f0288060280 2023-11-15T05:02:13.582 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.577+0000 7f0287fff700 1 --2- 172.21.15.178:0/1563269069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0288060280 0x7f02881066e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f027c013510 tx=0x7f027c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:13.583 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.577+0000 7f0284ff9700 1 -- 172.21.15.178:0/1563269069 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f027c009320 con 0x7f0288060280 2023-11-15T05:02:13.583 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.577+0000 7f028e6e3700 1 -- 172.21.15.178:0/1563269069 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0288106dd0 con 0x7f0288060280 2023-11-15T05:02:13.583 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.577+0000 7f0284ff9700 1 -- 172.21.15.178:0/1563269069 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f027c013f00 con 0x7f0288060280 2023-11-15T05:02:13.583 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.577+0000 7f0284ff9700 1 -- 172.21.15.178:0/1563269069 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f027c014dc0 con 0x7f0288060280 2023-11-15T05:02:13.583 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.577+0000 7f028e6e3700 1 -- 172.21.15.178:0/1563269069 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0288107260 con 0x7f0288060280 2023-11-15T05:02:13.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.577+0000 7f0284ff9700 1 -- 172.21.15.178:0/1563269069 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f027c014420 con 0x7f0288060280 2023-11-15T05:02:13.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.578+0000 7f028e6e3700 1 -- 172.21.15.178:0/1563269069 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0288045ad0 con 0x7f0288060280 2023-11-15T05:02:13.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.578+0000 7f0284ff9700 1 --2- 172.21.15.178:0/1563269069 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f027002ff60 0x7f0270032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:13.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.578+0000 7f0284ff9700 1 -- 172.21.15.178:0/1563269069 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f027c041030 con 0x7f0288060280 2023-11-15T05:02:13.586 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.578+0000 7f02877fe700 1 --2- 172.21.15.178:0/1563269069 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f027002ff60 0x7f0270032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:13.586 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.578+0000 7f02877fe700 1 --2- 172.21.15.178:0/1563269069 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f027002ff60 0x7f0270032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f02780096a0 tx=0x7f0278006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:13.586 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.583+0000 7f0284ff9700 1 -- 172.21.15.178:0/1563269069 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f027c0095f0 con 0x7f0288060280 2023-11-15T05:02:13.787 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.786+0000 7f028e6e3700 1 -- 172.21.15.178:0/1563269069 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0288108110 con 0x7f0288060280 2023-11-15T05:02:13.788 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.787+0000 7f0284ff9700 1 -- 172.21.15.178:0/1563269069 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f027c01cb10 con 0x7f0288060280 2023-11-15T05:02:13.789 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:02:13.789 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:13.790 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.788+0000 7f028e6e3700 1 -- 172.21.15.178:0/1563269069 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f027002ff60 msgr2=0x7f0270032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:13.790 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.788+0000 7f028e6e3700 1 --2- 172.21.15.178:0/1563269069 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f027002ff60 0x7f0270032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f02780096a0 tx=0x7f0278006b40).stop 2023-11-15T05:02:13.790 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.788+0000 7f028e6e3700 1 -- 172.21.15.178:0/1563269069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0288060280 msgr2=0x7f02881066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:13.791 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.788+0000 7f028e6e3700 1 --2- 172.21.15.178:0/1563269069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0288060280 0x7f02881066e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f027c013510 tx=0x7f027c00b040).stop 2023-11-15T05:02:13.791 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.788+0000 7f028e6e3700 1 -- 172.21.15.178:0/1563269069 shutdown_connections 2023-11-15T05:02:13.791 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.789+0000 7f028e6e3700 1 --2- 172.21.15.178:0/1563269069 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f027002ff60 0x7f0270032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:13.791 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.789+0000 7f028e6e3700 1 --2- 172.21.15.178:0/1563269069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0288060280 0x7f02881066e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:13.791 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.789+0000 7f028e6e3700 1 -- 172.21.15.178:0/1563269069 >> 172.21.15.178:0/1563269069 conn(0x7f02880f0b70 msgr2=0x7f02880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:13.791 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.789+0000 7f028e6e3700 1 -- 172.21.15.178:0/1563269069 shutdown_connections 2023-11-15T05:02:13.792 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:13.789+0000 7f028e6e3700 1 -- 172.21.15.178:0/1563269069 wait complete. 2023-11-15T05:02:13.792 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:14.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:14 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:12.372691+0000 mgr.smithi163.qesyrx (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:14.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:14 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:13.788658+0000 mon.smithi163 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.178:0/1563269069' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:15.179 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:15.179 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:16.378 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.375+0000 7f3e9b59e700 1 -- 172.21.15.178:0/1200440330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e9c0f51a0 msgr2=0x7f3e9c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:16.378 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.375+0000 7f3e9b59e700 1 --2- 172.21.15.178:0/1200440330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e9c0f51a0 0x7f3e9c0f55c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f3e84009fb0 tx=0x7f3e84009b30).stop 2023-11-15T05:02:16.378 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.375+0000 7f3e9b59e700 1 -- 172.21.15.178:0/1200440330 shutdown_connections 2023-11-15T05:02:16.378 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.375+0000 7f3e9b59e700 1 --2- 172.21.15.178:0/1200440330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e9c0f51a0 0x7f3e9c0f55c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:16.379 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.376+0000 7f3e9b59e700 1 -- 172.21.15.178:0/1200440330 >> 172.21.15.178:0/1200440330 conn(0x7f3e9c0f09a0 msgr2=0x7f3e9c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:16.379 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.376+0000 7f3e9b59e700 1 -- 172.21.15.178:0/1200440330 shutdown_connections 2023-11-15T05:02:16.379 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.376+0000 7f3e9b59e700 1 -- 172.21.15.178:0/1200440330 wait complete. 2023-11-15T05:02:16.379 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.376+0000 7f3e9b59e700 1 Processor -- start 2023-11-15T05:02:16.379 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.376+0000 7f3e9b59e700 1 -- start start 2023-11-15T05:02:16.380 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.376+0000 7f3e9b59e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e9c0f51a0 0x7f3e9c10b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:16.380 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.376+0000 7f3e9b59e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c106fc0 con 0x7f3e9c0f51a0 2023-11-15T05:02:16.380 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.376+0000 7f3e9a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e9c0f51a0 0x7f3e9c10b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:16.380 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.376+0000 7f3e9a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e9c0f51a0 0x7f3e9c10b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:53300/0 (socket says 172.21.15.178:53300) 2023-11-15T05:02:16.380 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.376+0000 7f3e9a59c700 1 -- 172.21.15.178:0/3323306974 learned_addr learned my addr 172.21.15.178:0/3323306974 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:02:16.381 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.377+0000 7f3e9a59c700 1 -- 172.21.15.178:0/3323306974 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e84005040 con 0x7f3e9c0f51a0 2023-11-15T05:02:16.381 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.377+0000 7f3e9a59c700 1 --2- 172.21.15.178:0/3323306974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e9c0f51a0 0x7f3e9c10b5d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f3e84013510 tx=0x7f3e8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:16.382 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.377+0000 7f3e92ffd700 1 -- 172.21.15.178:0/3323306974 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e84009320 con 0x7f3e9c0f51a0 2023-11-15T05:02:16.382 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.377+0000 7f3e9b59e700 1 -- 172.21.15.178:0/3323306974 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e9c060220 con 0x7f3e9c0f51a0 2023-11-15T05:02:16.382 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.377+0000 7f3e92ffd700 1 -- 172.21.15.178:0/3323306974 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e84013f00 con 0x7f3e9c0f51a0 2023-11-15T05:02:16.383 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.377+0000 7f3e92ffd700 1 -- 172.21.15.178:0/3323306974 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e84014dc0 con 0x7f3e9c0f51a0 2023-11-15T05:02:16.383 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.377+0000 7f3e9b59e700 1 -- 172.21.15.178:0/3323306974 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e9c10bac0 con 0x7f3e9c0f51a0 2023-11-15T05:02:16.383 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.378+0000 7f3e92ffd700 1 -- 172.21.15.178:0/3323306974 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3e840095f0 con 0x7f3e9c0f51a0 2023-11-15T05:02:16.383 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.378+0000 7f3e9b59e700 1 -- 172.21.15.178:0/3323306974 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e9c045ad0 con 0x7f3e9c0f51a0 2023-11-15T05:02:16.383 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.378+0000 7f3e92ffd700 1 --2- 172.21.15.178:0/3323306974 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3e8802ff60 0x7f3e88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:16.384 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.378+0000 7f3e92ffd700 1 -- 172.21.15.178:0/3323306974 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3e84041040 con 0x7f3e9c0f51a0 2023-11-15T05:02:16.384 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.378+0000 7f3e99d9b700 1 --2- 172.21.15.178:0/3323306974 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3e8802ff60 0x7f3e88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:16.384 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.379+0000 7f3e99d9b700 1 --2- 172.21.15.178:0/3323306974 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3e8802ff60 0x7f3e88032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f3e8c0096a0 tx=0x7f3e8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:16.384 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.383+0000 7f3e92ffd700 1 -- 172.21.15.178:0/3323306974 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3e84014420 con 0x7f3e9c0f51a0 2023-11-15T05:02:16.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:14.372971+0000 mgr.smithi163.qesyrx (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:16.588 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.587+0000 7f3e9b59e700 1 -- 172.21.15.178:0/3323306974 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e9c10c280 con 0x7f3e9c0f51a0 2023-11-15T05:02:16.589 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.588+0000 7f3e92ffd700 1 -- 172.21.15.178:0/3323306974 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3e84010230 con 0x7f3e9c0f51a0 2023-11-15T05:02:16.590 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:02:16.590 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:16.592 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.589+0000 7f3e9b59e700 1 -- 172.21.15.178:0/3323306974 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3e8802ff60 msgr2=0x7f3e88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:16.592 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.589+0000 7f3e9b59e700 1 --2- 172.21.15.178:0/3323306974 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3e8802ff60 0x7f3e88032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f3e8c0096a0 tx=0x7f3e8c006b40).stop 2023-11-15T05:02:16.592 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.589+0000 7f3e9b59e700 1 -- 172.21.15.178:0/3323306974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e9c0f51a0 msgr2=0x7f3e9c10b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:16.593 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.589+0000 7f3e9b59e700 1 --2- 172.21.15.178:0/3323306974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e9c0f51a0 0x7f3e9c10b5d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f3e84013510 tx=0x7f3e8400b040).stop 2023-11-15T05:02:16.593 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.589+0000 7f3e9b59e700 1 -- 172.21.15.178:0/3323306974 shutdown_connections 2023-11-15T05:02:16.593 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.590+0000 7f3e9b59e700 1 --2- 172.21.15.178:0/3323306974 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3e8802ff60 0x7f3e88032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:16.593 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.590+0000 7f3e9b59e700 1 --2- 172.21.15.178:0/3323306974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e9c0f51a0 0x7f3e9c10b5d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:16.593 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.590+0000 7f3e9b59e700 1 -- 172.21.15.178:0/3323306974 >> 172.21.15.178:0/3323306974 conn(0x7f3e9c0f09a0 msgr2=0x7f3e9c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:16.594 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.590+0000 7f3e9b59e700 1 -- 172.21.15.178:0/3323306974 shutdown_connections 2023-11-15T05:02:16.594 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:16.590+0000 7f3e9b59e700 1 -- 172.21.15.178:0/3323306974 wait complete. 2023-11-15T05:02:16.594 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:17.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:17 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:16.589469+0000 mon.smithi163 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.178:0/3323306974' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:17.979 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:17.979 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:18.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:18 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:16.373326+0000 mgr.smithi163.qesyrx (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:19.137 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.136+0000 7fdc75121700 1 -- 172.21.15.178:0/402407550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc700f5410 msgr2=0x7fdc700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:19.138 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.136+0000 7fdc75121700 1 --2- 172.21.15.178:0/402407550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc700f5410 0x7fdc700f5830 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fdc64009fb0 tx=0x7fdc64009b30).stop 2023-11-15T05:02:19.138 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.136+0000 7fdc75121700 1 -- 172.21.15.178:0/402407550 shutdown_connections 2023-11-15T05:02:19.138 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.136+0000 7fdc75121700 1 --2- 172.21.15.178:0/402407550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc700f5410 0x7fdc700f5830 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:19.138 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.136+0000 7fdc75121700 1 -- 172.21.15.178:0/402407550 >> 172.21.15.178:0/402407550 conn(0x7fdc700f0b70 msgr2=0x7fdc700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:19.138 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.136+0000 7fdc75121700 1 -- 172.21.15.178:0/402407550 shutdown_connections 2023-11-15T05:02:19.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.136+0000 7fdc75121700 1 -- 172.21.15.178:0/402407550 wait complete. 2023-11-15T05:02:19.141 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.137+0000 7fdc75121700 1 Processor -- start 2023-11-15T05:02:19.141 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.137+0000 7fdc75121700 1 -- start start 2023-11-15T05:02:19.141 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.137+0000 7fdc75121700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc700f5410 0x7fdc70109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:19.141 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.137+0000 7fdc75121700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc70060220 con 0x7fdc700f5410 2023-11-15T05:02:19.141 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.137+0000 7fdc6ed9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc700f5410 0x7fdc70109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:19.142 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.137+0000 7fdc6ed9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc700f5410 0x7fdc70109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:53326/0 (socket says 172.21.15.178:53326) 2023-11-15T05:02:19.142 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.137+0000 7fdc6ed9d700 1 -- 172.21.15.178:0/3005676059 learned_addr learned my addr 172.21.15.178:0/3005676059 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:02:19.142 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.138+0000 7fdc6ed9d700 1 -- 172.21.15.178:0/3005676059 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc64005040 con 0x7fdc700f5410 2023-11-15T05:02:19.142 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.138+0000 7fdc6ed9d700 1 --2- 172.21.15.178:0/3005676059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc700f5410 0x7fdc70109450 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fdc64014000 tx=0x7fdc6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:19.142 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.138+0000 7fdc577fe700 1 -- 172.21.15.178:0/3005676059 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc640095f0 con 0x7fdc700f5410 2023-11-15T05:02:19.143 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.138+0000 7fdc75121700 1 -- 172.21.15.178:0/3005676059 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc70060420 con 0x7fdc700f5410 2023-11-15T05:02:19.143 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.138+0000 7fdc577fe700 1 -- 172.21.15.178:0/3005676059 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc64007e70 con 0x7fdc700f5410 2023-11-15T05:02:19.143 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.138+0000 7fdc577fe700 1 -- 172.21.15.178:0/3005676059 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc64014ea0 con 0x7fdc700f5410 2023-11-15T05:02:19.143 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.138+0000 7fdc75121700 1 -- 172.21.15.178:0/3005676059 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc70109b00 con 0x7fdc700f5410 2023-11-15T05:02:19.143 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.139+0000 7fdc577fe700 1 -- 172.21.15.178:0/3005676059 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc64025070 con 0x7fdc700f5410 2023-11-15T05:02:19.144 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.139+0000 7fdc75121700 1 -- 172.21.15.178:0/3005676059 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc70059100 con 0x7fdc700f5410 2023-11-15T05:02:19.144 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.139+0000 7fdc577fe700 1 --2- 172.21.15.178:0/3005676059 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdc5802ff60 0x7fdc58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:19.144 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.139+0000 7fdc577fe700 1 -- 172.21.15.178:0/3005676059 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc64040940 con 0x7fdc700f5410 2023-11-15T05:02:19.144 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.139+0000 7fdc6e59c700 1 --2- 172.21.15.178:0/3005676059 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdc5802ff60 0x7fdc58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:19.144 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.139+0000 7fdc6e59c700 1 --2- 172.21.15.178:0/3005676059 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdc5802ff60 0x7fdc58032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fdc600096a0 tx=0x7fdc60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:19.145 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.144+0000 7fdc577fe700 1 -- 172.21.15.178:0/3005676059 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc640079c0 con 0x7fdc700f5410 2023-11-15T05:02:19.351 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.349+0000 7fdc75121700 1 -- 172.21.15.178:0/3005676059 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc70045ad0 con 0x7fdc700f5410 2023-11-15T05:02:19.351 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.350+0000 7fdc577fe700 1 -- 172.21.15.178:0/3005676059 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdc6400f300 con 0x7fdc700f5410 2023-11-15T05:02:19.352 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:02:19.352 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:19.353 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.351+0000 7fdc75121700 1 -- 172.21.15.178:0/3005676059 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdc5802ff60 msgr2=0x7fdc58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:19.353 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.351+0000 7fdc75121700 1 --2- 172.21.15.178:0/3005676059 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdc5802ff60 0x7fdc58032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fdc600096a0 tx=0x7fdc60006b40).stop 2023-11-15T05:02:19.353 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.351+0000 7fdc75121700 1 -- 172.21.15.178:0/3005676059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc700f5410 msgr2=0x7fdc70109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:19.354 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.351+0000 7fdc75121700 1 --2- 172.21.15.178:0/3005676059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc700f5410 0x7fdc70109450 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fdc64014000 tx=0x7fdc6400b040).stop 2023-11-15T05:02:19.354 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.351+0000 7fdc75121700 1 -- 172.21.15.178:0/3005676059 shutdown_connections 2023-11-15T05:02:19.354 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.351+0000 7fdc75121700 1 --2- 172.21.15.178:0/3005676059 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdc5802ff60 0x7fdc58032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:19.354 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.351+0000 7fdc75121700 1 --2- 172.21.15.178:0/3005676059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc700f5410 0x7fdc70109450 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:19.355 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.351+0000 7fdc75121700 1 -- 172.21.15.178:0/3005676059 >> 172.21.15.178:0/3005676059 conn(0x7fdc700f0b70 msgr2=0x7fdc700f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:19.355 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.351+0000 7fdc75121700 1 -- 172.21.15.178:0/3005676059 shutdown_connections 2023-11-15T05:02:19.355 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:19.351+0000 7fdc75121700 1 -- 172.21.15.178:0/3005676059 wait complete. 2023-11-15T05:02:19.356 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:20.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:20 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:18.373719+0000 mgr.smithi163.qesyrx (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:20.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:20 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:19.351647+0000 mon.smithi163 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.178:0/3005676059' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:20.781 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:20.781 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:21.959 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.957+0000 7f986975e700 1 -- 172.21.15.178:0/3655906176 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98640f6510 msgr2=0x7f98640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:21.959 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.957+0000 7f986975e700 1 --2- 172.21.15.178:0/3655906176 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98640f6510 0x7f98640f6930 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f9858009fb0 tx=0x7f9858009b30).stop 2023-11-15T05:02:21.960 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.957+0000 7f986975e700 1 -- 172.21.15.178:0/3655906176 shutdown_connections 2023-11-15T05:02:21.960 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.957+0000 7f986975e700 1 --2- 172.21.15.178:0/3655906176 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98640f6510 0x7f98640f6930 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:21.960 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.957+0000 7f986975e700 1 -- 172.21.15.178:0/3655906176 >> 172.21.15.178:0/3655906176 conn(0x7f98640f1c70 msgr2=0x7f98640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:21.960 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.957+0000 7f986975e700 1 -- 172.21.15.178:0/3655906176 shutdown_connections 2023-11-15T05:02:21.960 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.957+0000 7f986975e700 1 -- 172.21.15.178:0/3655906176 wait complete. 2023-11-15T05:02:21.961 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.958+0000 7f986975e700 1 Processor -- start 2023-11-15T05:02:21.961 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.958+0000 7f986975e700 1 -- start start 2023-11-15T05:02:21.961 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.958+0000 7f986975e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98640f6510 0x7f98641066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:21.961 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.958+0000 7f986975e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9864106bc0 con 0x7f98640f6510 2023-11-15T05:02:21.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.958+0000 7f9862ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98640f6510 0x7f98641066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:21.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.958+0000 7f9862ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98640f6510 0x7f98641066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:53350/0 (socket says 172.21.15.178:53350) 2023-11-15T05:02:21.963 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.958+0000 7f9862ffd700 1 -- 172.21.15.178:0/3692884413 learned_addr learned my addr 172.21.15.178:0/3692884413 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:02:21.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.959+0000 7f9862ffd700 1 -- 172.21.15.178:0/3692884413 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9858005040 con 0x7f98640f6510 2023-11-15T05:02:21.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.959+0000 7f9862ffd700 1 --2- 172.21.15.178:0/3692884413 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98640f6510 0x7f98641066d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f9858013510 tx=0x7f985800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:21.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.959+0000 7f984b7fe700 1 -- 172.21.15.178:0/3692884413 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9858009320 con 0x7f98640f6510 2023-11-15T05:02:21.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.959+0000 7f986975e700 1 -- 172.21.15.178:0/3692884413 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9864106dc0 con 0x7f98640f6510 2023-11-15T05:02:21.964 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.959+0000 7f984b7fe700 1 -- 172.21.15.178:0/3692884413 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9858013f00 con 0x7f98640f6510 2023-11-15T05:02:21.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.959+0000 7f984b7fe700 1 -- 172.21.15.178:0/3692884413 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9858014d90 con 0x7f98640f6510 2023-11-15T05:02:21.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.959+0000 7f986975e700 1 -- 172.21.15.178:0/3692884413 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9864107250 con 0x7f98640f6510 2023-11-15T05:02:21.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.960+0000 7f984b7fe700 1 -- 172.21.15.178:0/3692884413 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f98580095f0 con 0x7f98640f6510 2023-11-15T05:02:21.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.960+0000 7f986975e700 1 -- 172.21.15.178:0/3692884413 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9864045ad0 con 0x7f98640f6510 2023-11-15T05:02:21.965 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.960+0000 7f984b7fe700 1 --2- 172.21.15.178:0/3692884413 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f984c02ff60 0x7f984c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:21.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.960+0000 7f984b7fe700 1 -- 172.21.15.178:0/3692884413 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9858040fe0 con 0x7f98640f6510 2023-11-15T05:02:21.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.960+0000 7f98627fc700 1 --2- 172.21.15.178:0/3692884413 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f984c02ff60 0x7f984c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:21.966 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.960+0000 7f98627fc700 1 --2- 172.21.15.178:0/3692884413 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f984c02ff60 0x7f984c032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f98540096a0 tx=0x7f9854006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:21.967 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:21.965+0000 7f984b7fe700 1 -- 172.21.15.178:0/3692884413 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f985801d7f0 con 0x7f98640f6510 2023-11-15T05:02:22.172 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:22.171+0000 7f986975e700 1 -- 172.21.15.178:0/3692884413 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9864108100 con 0x7f98640f6510 2023-11-15T05:02:22.173 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:22.172+0000 7f984b7fe700 1 -- 172.21.15.178:0/3692884413 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9858010230 con 0x7f98640f6510 2023-11-15T05:02:22.174 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:02:22.174 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:22.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:22.174+0000 7f986975e700 1 -- 172.21.15.178:0/3692884413 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f984c02ff60 msgr2=0x7f984c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:22.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:22.174+0000 7f986975e700 1 --2- 172.21.15.178:0/3692884413 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f984c02ff60 0x7f984c032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f98540096a0 tx=0x7f9854006b40).stop 2023-11-15T05:02:22.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:22.174+0000 7f986975e700 1 -- 172.21.15.178:0/3692884413 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98640f6510 msgr2=0x7f98641066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:22.175 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:22.174+0000 7f986975e700 1 --2- 172.21.15.178:0/3692884413 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98640f6510 0x7f98641066d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f9858013510 tx=0x7f985800b040).stop 2023-11-15T05:02:22.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:22.174+0000 7f986975e700 1 -- 172.21.15.178:0/3692884413 shutdown_connections 2023-11-15T05:02:22.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:22.174+0000 7f986975e700 1 --2- 172.21.15.178:0/3692884413 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f984c02ff60 0x7f984c032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:22.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:22.174+0000 7f986975e700 1 --2- 172.21.15.178:0/3692884413 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98640f6510 0x7f98641066d0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:22.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:22.174+0000 7f986975e700 1 -- 172.21.15.178:0/3692884413 >> 172.21.15.178:0/3692884413 conn(0x7f98640f1c70 msgr2=0x7f98640f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:22.176 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:22.174+0000 7f986975e700 1 -- 172.21.15.178:0/3692884413 shutdown_connections 2023-11-15T05:02:22.177 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:22.174+0000 7f986975e700 1 -- 172.21.15.178:0/3692884413 wait complete. 2023-11-15T05:02:22.177 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:22.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:22 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:20.374098+0000 mgr.smithi163.qesyrx (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:23.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:23 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:22.173741+0000 mon.smithi163 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.178:0/3692884413' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:23.558 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:23.559 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:24.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:24 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:22.374393+0000 mgr.smithi163.qesyrx (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:24.795 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.793+0000 7fa3769bd700 1 -- 172.21.15.178:0/473230828 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa3700f6510 msgr2=0x7fa3700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:24.795 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.793+0000 7fa3769bd700 1 --2- 172.21.15.178:0/473230828 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa3700f6510 0x7fa3700f6930 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fa360009fb0 tx=0x7fa360009b30).stop 2023-11-15T05:02:24.795 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.793+0000 7fa3769bd700 1 -- 172.21.15.178:0/473230828 shutdown_connections 2023-11-15T05:02:24.796 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.793+0000 7fa3769bd700 1 --2- 172.21.15.178:0/473230828 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa3700f6510 0x7fa3700f6930 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:24.796 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.793+0000 7fa3769bd700 1 -- 172.21.15.178:0/473230828 >> 172.21.15.178:0/473230828 conn(0x7fa3700f1c70 msgr2=0x7fa3700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:24.796 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.793+0000 7fa3769bd700 1 -- 172.21.15.178:0/473230828 shutdown_connections 2023-11-15T05:02:24.796 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.793+0000 7fa3769bd700 1 -- 172.21.15.178:0/473230828 wait complete. 2023-11-15T05:02:24.798 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.794+0000 7fa3769bd700 1 Processor -- start 2023-11-15T05:02:24.798 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.794+0000 7fa3769bd700 1 -- start start 2023-11-15T05:02:24.798 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.795+0000 7fa3769bd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa3700f6510 0x7fa370106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:24.799 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.795+0000 7fa3769bd700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa370106c20 con 0x7fa3700f6510 2023-11-15T05:02:24.799 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.795+0000 7fa36ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa3700f6510 0x7fa370106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:24.799 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.795+0000 7fa36ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa3700f6510 0x7fa370106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:35610/0 (socket says 172.21.15.178:35610) 2023-11-15T05:02:24.799 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.795+0000 7fa36ffff700 1 -- 172.21.15.178:0/1615662990 learned_addr learned my addr 172.21.15.178:0/1615662990 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:02:24.799 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.795+0000 7fa36ffff700 1 -- 172.21.15.178:0/1615662990 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa360005040 con 0x7fa3700f6510 2023-11-15T05:02:24.800 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.795+0000 7fa36ffff700 1 --2- 172.21.15.178:0/1615662990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa3700f6510 0x7fa370106730 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fa360013510 tx=0x7fa36000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:24.800 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.795+0000 7fa36cff9700 1 -- 172.21.15.178:0/1615662990 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa360009320 con 0x7fa3700f6510 2023-11-15T05:02:24.800 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.795+0000 7fa36cff9700 1 -- 172.21.15.178:0/1615662990 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa360013f00 con 0x7fa3700f6510 2023-11-15T05:02:24.800 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.795+0000 7fa3769bd700 1 -- 172.21.15.178:0/1615662990 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa370106e20 con 0x7fa3700f6510 2023-11-15T05:02:24.800 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.795+0000 7fa36cff9700 1 -- 172.21.15.178:0/1615662990 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa360014d90 con 0x7fa3700f6510 2023-11-15T05:02:24.801 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.795+0000 7fa3769bd700 1 -- 172.21.15.178:0/1615662990 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3701072b0 con 0x7fa3700f6510 2023-11-15T05:02:24.801 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.796+0000 7fa36cff9700 1 -- 172.21.15.178:0/1615662990 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3600095f0 con 0x7fa3700f6510 2023-11-15T05:02:24.802 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.796+0000 7fa3769bd700 1 -- 172.21.15.178:0/1615662990 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa370045ad0 con 0x7fa3700f6510 2023-11-15T05:02:24.802 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.796+0000 7fa36cff9700 1 --2- 172.21.15.178:0/1615662990 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa35802ff60 0x7fa358032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:24.802 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.796+0000 7fa36cff9700 1 -- 172.21.15.178:0/1615662990 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3600410c0 con 0x7fa3700f6510 2023-11-15T05:02:24.802 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.796+0000 7fa36f7fe700 1 --2- 172.21.15.178:0/1615662990 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa35802ff60 0x7fa358032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:24.802 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.797+0000 7fa36f7fe700 1 --2- 172.21.15.178:0/1615662990 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa35802ff60 0x7fa358032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fa3640096a0 tx=0x7fa364006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:24.804 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:24.802+0000 7fa36cff9700 1 -- 172.21.15.178:0/1615662990 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa360014420 con 0x7fa3700f6510 2023-11-15T05:02:25.006 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:25.004+0000 7fa3769bd700 1 -- 172.21.15.178:0/1615662990 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3700fbf60 con 0x7fa3700f6510 2023-11-15T05:02:25.006 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:25.005+0000 7fa36cff9700 1 -- 172.21.15.178:0/1615662990 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa360010230 con 0x7fa3700f6510 2023-11-15T05:02:25.007 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:02:25.007 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:25.008 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:25.007+0000 7fa3769bd700 1 -- 172.21.15.178:0/1615662990 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa35802ff60 msgr2=0x7fa358032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:25.008 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:25.007+0000 7fa3769bd700 1 --2- 172.21.15.178:0/1615662990 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa35802ff60 0x7fa358032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fa3640096a0 tx=0x7fa364006b40).stop 2023-11-15T05:02:25.008 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:25.007+0000 7fa3769bd700 1 -- 172.21.15.178:0/1615662990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa3700f6510 msgr2=0x7fa370106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:25.009 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:25.007+0000 7fa3769bd700 1 --2- 172.21.15.178:0/1615662990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa3700f6510 0x7fa370106730 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fa360013510 tx=0x7fa36000b040).stop 2023-11-15T05:02:25.009 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:25.007+0000 7fa3769bd700 1 -- 172.21.15.178:0/1615662990 shutdown_connections 2023-11-15T05:02:25.009 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:25.007+0000 7fa3769bd700 1 --2- 172.21.15.178:0/1615662990 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa35802ff60 0x7fa358032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:25.009 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:25.007+0000 7fa3769bd700 1 --2- 172.21.15.178:0/1615662990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa3700f6510 0x7fa370106730 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:25.009 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:25.007+0000 7fa3769bd700 1 -- 172.21.15.178:0/1615662990 >> 172.21.15.178:0/1615662990 conn(0x7fa3700f1c70 msgr2=0x7fa3700f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:25.010 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:25.007+0000 7fa3769bd700 1 -- 172.21.15.178:0/1615662990 shutdown_connections 2023-11-15T05:02:25.010 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:25.007+0000 7fa3769bd700 1 -- 172.21.15.178:0/1615662990 wait complete. 2023-11-15T05:02:25.012 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:25.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:24.454862+0000 mon.smithi163 (mon.0) 276 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:02:25.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:24.455833+0000 mon.smithi163 (mon.0) 277 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/trash_purge_schedule"}]: dispatch 2023-11-15T05:02:25.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:25.006751+0000 mon.smithi163 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.178:0/1615662990' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:26.406 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:26.407 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:26.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:26 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:24.374698+0000 mgr.smithi163.qesyrx (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:27.594 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.593+0000 7f0e4a0b2700 1 -- 172.21.15.178:0/695631750 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e44060280 msgr2=0x7f0e440f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:27.594 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.593+0000 7f0e4a0b2700 1 --2- 172.21.15.178:0/695631750 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e44060280 0x7f0e440f90f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f0e34009fb0 tx=0x7f0e34009b30).stop 2023-11-15T05:02:27.594 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.593+0000 7f0e4a0b2700 1 -- 172.21.15.178:0/695631750 shutdown_connections 2023-11-15T05:02:27.595 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.593+0000 7f0e4a0b2700 1 --2- 172.21.15.178:0/695631750 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e44060280 0x7f0e440f90f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:27.595 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.593+0000 7f0e4a0b2700 1 -- 172.21.15.178:0/695631750 >> 172.21.15.178:0/695631750 conn(0x7f0e440f0b70 msgr2=0x7f0e440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:27.595 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.593+0000 7f0e4a0b2700 1 -- 172.21.15.178:0/695631750 shutdown_connections 2023-11-15T05:02:27.595 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.593+0000 7f0e4a0b2700 1 -- 172.21.15.178:0/695631750 wait complete. 2023-11-15T05:02:27.595 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.594+0000 7f0e4a0b2700 1 Processor -- start 2023-11-15T05:02:27.598 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.594+0000 7f0e4a0b2700 1 -- start start 2023-11-15T05:02:27.598 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.594+0000 7f0e4a0b2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e44060280 0x7f0e441066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:27.598 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.594+0000 7f0e4a0b2700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e44106bd0 con 0x7f0e44060280 2023-11-15T05:02:27.598 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.594+0000 7f0e437fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e44060280 0x7f0e441066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:27.599 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.594+0000 7f0e437fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e44060280 0x7f0e441066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:35624/0 (socket says 172.21.15.178:35624) 2023-11-15T05:02:27.599 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.594+0000 7f0e437fe700 1 -- 172.21.15.178:0/116421257 learned_addr learned my addr 172.21.15.178:0/116421257 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:02:27.599 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.595+0000 7f0e437fe700 1 -- 172.21.15.178:0/116421257 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e34005040 con 0x7f0e44060280 2023-11-15T05:02:27.599 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.595+0000 7f0e437fe700 1 --2- 172.21.15.178:0/116421257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e44060280 0x7f0e441066e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f0e34013510 tx=0x7f0e3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:27.600 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.595+0000 7f0e2bfff700 1 -- 172.21.15.178:0/116421257 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e34009320 con 0x7f0e44060280 2023-11-15T05:02:27.600 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.595+0000 7f0e4a0b2700 1 -- 172.21.15.178:0/116421257 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e44106dd0 con 0x7f0e44060280 2023-11-15T05:02:27.600 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.595+0000 7f0e2bfff700 1 -- 172.21.15.178:0/116421257 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e34013f00 con 0x7f0e44060280 2023-11-15T05:02:27.600 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.595+0000 7f0e2bfff700 1 -- 172.21.15.178:0/116421257 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e34014dc0 con 0x7f0e44060280 2023-11-15T05:02:27.600 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.595+0000 7f0e4a0b2700 1 -- 172.21.15.178:0/116421257 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e44107260 con 0x7f0e44060280 2023-11-15T05:02:27.601 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.596+0000 7f0e2bfff700 1 -- 172.21.15.178:0/116421257 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0e340095f0 con 0x7f0e44060280 2023-11-15T05:02:27.601 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.596+0000 7f0e4a0b2700 1 -- 172.21.15.178:0/116421257 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e44045ad0 con 0x7f0e44060280 2023-11-15T05:02:27.601 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.596+0000 7f0e2bfff700 1 --2- 172.21.15.178:0/116421257 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f0e2c02ff60 0x7f0e2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:27.601 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.596+0000 7f0e2bfff700 1 -- 172.21.15.178:0/116421257 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0e34041040 con 0x7f0e44060280 2023-11-15T05:02:27.601 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.596+0000 7f0e42ffd700 1 --2- 172.21.15.178:0/116421257 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f0e2c02ff60 0x7f0e2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:27.602 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.597+0000 7f0e42ffd700 1 --2- 172.21.15.178:0/116421257 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f0e2c02ff60 0x7f0e2c032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f0e380096a0 tx=0x7f0e38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:27.602 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.601+0000 7f0e2bfff700 1 -- 172.21.15.178:0/116421257 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e34014420 con 0x7f0e44060280 2023-11-15T05:02:27.806 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.805+0000 7f0e4a0b2700 1 -- 172.21.15.178:0/116421257 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e44108110 con 0x7f0e44060280 2023-11-15T05:02:27.806 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.806+0000 7f0e2bfff700 1 -- 172.21.15.178:0/116421257 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0e34010230 con 0x7f0e44060280 2023-11-15T05:02:27.807 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:02:27.807 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:27.808 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.808+0000 7f0e4a0b2700 1 -- 172.21.15.178:0/116421257 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f0e2c02ff60 msgr2=0x7f0e2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:27.808 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.808+0000 7f0e4a0b2700 1 --2- 172.21.15.178:0/116421257 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f0e2c02ff60 0x7f0e2c032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f0e380096a0 tx=0x7f0e38006b40).stop 2023-11-15T05:02:27.808 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.808+0000 7f0e4a0b2700 1 -- 172.21.15.178:0/116421257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e44060280 msgr2=0x7f0e441066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:27.809 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.808+0000 7f0e4a0b2700 1 --2- 172.21.15.178:0/116421257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e44060280 0x7f0e441066e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f0e34013510 tx=0x7f0e3400b040).stop 2023-11-15T05:02:27.809 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.808+0000 7f0e4a0b2700 1 -- 172.21.15.178:0/116421257 shutdown_connections 2023-11-15T05:02:27.809 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.808+0000 7f0e4a0b2700 1 --2- 172.21.15.178:0/116421257 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f0e2c02ff60 0x7f0e2c032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:27.809 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.808+0000 7f0e4a0b2700 1 --2- 172.21.15.178:0/116421257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e44060280 0x7f0e441066e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:27.809 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.808+0000 7f0e4a0b2700 1 -- 172.21.15.178:0/116421257 >> 172.21.15.178:0/116421257 conn(0x7f0e440f0b70 msgr2=0x7f0e440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:27.810 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.808+0000 7f0e4a0b2700 1 -- 172.21.15.178:0/116421257 shutdown_connections 2023-11-15T05:02:27.810 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:27.808+0000 7f0e4a0b2700 1 -- 172.21.15.178:0/116421257 wait complete. 2023-11-15T05:02:27.810 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:28.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:28 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:26.375058+0000 mgr.smithi163.qesyrx (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:28.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:28 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:27.806775+0000 mon.smithi163 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.178:0/116421257' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:29.161 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:29.162 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:30.336 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.334+0000 7f8140ac2700 1 -- 172.21.15.178:0/2808773662 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f813c0f5410 msgr2=0x7f813c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:30.336 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.334+0000 7f8140ac2700 1 --2- 172.21.15.178:0/2808773662 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f813c0f5410 0x7f813c0f5830 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f812c009fb0 tx=0x7f812c009b30).stop 2023-11-15T05:02:30.336 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.335+0000 7f8140ac2700 1 -- 172.21.15.178:0/2808773662 shutdown_connections 2023-11-15T05:02:30.337 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.335+0000 7f8140ac2700 1 --2- 172.21.15.178:0/2808773662 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f813c0f5410 0x7f813c0f5830 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:30.337 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.335+0000 7f8140ac2700 1 -- 172.21.15.178:0/2808773662 >> 172.21.15.178:0/2808773662 conn(0x7f813c0f0b70 msgr2=0x7f813c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:30.337 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.335+0000 7f8140ac2700 1 -- 172.21.15.178:0/2808773662 shutdown_connections 2023-11-15T05:02:30.337 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.335+0000 7f8140ac2700 1 -- 172.21.15.178:0/2808773662 wait complete. 2023-11-15T05:02:30.337 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.335+0000 7f8140ac2700 1 Processor -- start 2023-11-15T05:02:30.338 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.335+0000 7f8140ac2700 1 -- start start 2023-11-15T05:02:30.338 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.335+0000 7f8140ac2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f813c0f5410 0x7f813c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:30.338 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.335+0000 7f8140ac2700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c106c10 con 0x7f813c0f5410 2023-11-15T05:02:30.338 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.336+0000 7f813a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f813c0f5410 0x7f813c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:30.338 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.336+0000 7f813a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f813c0f5410 0x7f813c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:35648/0 (socket says 172.21.15.178:35648) 2023-11-15T05:02:30.339 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.336+0000 7f813a59c700 1 -- 172.21.15.178:0/1249013835 learned_addr learned my addr 172.21.15.178:0/1249013835 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:02:30.339 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.336+0000 7f813a59c700 1 -- 172.21.15.178:0/1249013835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812c005040 con 0x7f813c0f5410 2023-11-15T05:02:30.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.336+0000 7f813a59c700 1 --2- 172.21.15.178:0/1249013835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f813c0f5410 0x7f813c106720 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f812c013510 tx=0x7f812c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:30.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.336+0000 7f8132ffd700 1 -- 172.21.15.178:0/1249013835 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f812c009320 con 0x7f813c0f5410 2023-11-15T05:02:30.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.336+0000 7f8140ac2700 1 -- 172.21.15.178:0/1249013835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813c106e10 con 0x7f813c0f5410 2023-11-15T05:02:30.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.336+0000 7f8132ffd700 1 -- 172.21.15.178:0/1249013835 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f812c013f00 con 0x7f813c0f5410 2023-11-15T05:02:30.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.336+0000 7f8132ffd700 1 -- 172.21.15.178:0/1249013835 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f812c014d90 con 0x7f813c0f5410 2023-11-15T05:02:30.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.336+0000 7f8140ac2700 1 -- 172.21.15.178:0/1249013835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813c1072a0 con 0x7f813c0f5410 2023-11-15T05:02:30.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.337+0000 7f8132ffd700 1 -- 172.21.15.178:0/1249013835 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f812c0263a0 con 0x7f813c0f5410 2023-11-15T05:02:30.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.337+0000 7f8140ac2700 1 -- 172.21.15.178:0/1249013835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813c045ad0 con 0x7f813c0f5410 2023-11-15T05:02:30.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.337+0000 7f8132ffd700 1 --2- 172.21.15.178:0/1249013835 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f812802ff60 0x7f8128032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:30.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.337+0000 7f8132ffd700 1 -- 172.21.15.178:0/1249013835 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f812c0412d0 con 0x7f813c0f5410 2023-11-15T05:02:30.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.337+0000 7f8139d9b700 1 --2- 172.21.15.178:0/1249013835 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f812802ff60 0x7f8128032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:30.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.338+0000 7f8139d9b700 1 --2- 172.21.15.178:0/1249013835 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f812802ff60 0x7f8128032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f81240096a0 tx=0x7f8124006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:30.343 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.342+0000 7f8132ffd700 1 -- 172.21.15.178:0/1249013835 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f812c01cb10 con 0x7f813c0f5410 2023-11-15T05:02:30.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:30 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:28.375326+0000 mgr.smithi163.qesyrx (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:30.551 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.551+0000 7f8140ac2700 1 -- 172.21.15.178:0/1249013835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f813c108150 con 0x7f813c0f5410 2023-11-15T05:02:30.552 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.552+0000 7f8132ffd700 1 -- 172.21.15.178:0/1249013835 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f812c01cb10 con 0x7f813c0f5410 2023-11-15T05:02:30.552 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:02:30.552 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:30.553 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.553+0000 7f8140ac2700 1 -- 172.21.15.178:0/1249013835 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f812802ff60 msgr2=0x7f8128032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:30.554 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.553+0000 7f8140ac2700 1 --2- 172.21.15.178:0/1249013835 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f812802ff60 0x7f8128032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f81240096a0 tx=0x7f8124006b40).stop 2023-11-15T05:02:30.554 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.553+0000 7f8140ac2700 1 -- 172.21.15.178:0/1249013835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f813c0f5410 msgr2=0x7f813c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:30.554 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.553+0000 7f8140ac2700 1 --2- 172.21.15.178:0/1249013835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f813c0f5410 0x7f813c106720 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f812c013510 tx=0x7f812c00b040).stop 2023-11-15T05:02:30.554 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.553+0000 7f8140ac2700 1 -- 172.21.15.178:0/1249013835 shutdown_connections 2023-11-15T05:02:30.555 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.553+0000 7f8140ac2700 1 --2- 172.21.15.178:0/1249013835 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f812802ff60 0x7f8128032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:30.555 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.553+0000 7f8140ac2700 1 --2- 172.21.15.178:0/1249013835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f813c0f5410 0x7f813c106720 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:30.555 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.553+0000 7f8140ac2700 1 -- 172.21.15.178:0/1249013835 >> 172.21.15.178:0/1249013835 conn(0x7f813c0f0b70 msgr2=0x7f813c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:30.555 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.553+0000 7f8140ac2700 1 -- 172.21.15.178:0/1249013835 shutdown_connections 2023-11-15T05:02:30.558 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:30.553+0000 7f8140ac2700 1 -- 172.21.15.178:0/1249013835 wait complete. 2023-11-15T05:02:30.558 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:31.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:31 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:30.552336+0000 mon.smithi163 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.178:0/1249013835' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:31.948 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:31.949 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:32.454 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:32 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:30.375669+0000 mgr.smithi163.qesyrx (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:33.124 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.122+0000 7f1fa526c700 1 -- 172.21.15.178:0/1007332629 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fa00f59d0 msgr2=0x7f1fa00f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:33.124 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.122+0000 7f1fa526c700 1 --2- 172.21.15.178:0/1007332629 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fa00f59d0 0x7f1fa00f5df0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f1f94009fb0 tx=0x7f1f94009b30).stop 2023-11-15T05:02:33.125 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.122+0000 7f1fa526c700 1 -- 172.21.15.178:0/1007332629 shutdown_connections 2023-11-15T05:02:33.125 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.122+0000 7f1fa526c700 1 --2- 172.21.15.178:0/1007332629 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fa00f59d0 0x7f1fa00f5df0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:33.125 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.122+0000 7f1fa526c700 1 -- 172.21.15.178:0/1007332629 >> 172.21.15.178:0/1007332629 conn(0x7f1fa00f02e0 msgr2=0x7f1fa00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:33.125 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.122+0000 7f1fa526c700 1 -- 172.21.15.178:0/1007332629 shutdown_connections 2023-11-15T05:02:33.126 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.122+0000 7f1fa526c700 1 -- 172.21.15.178:0/1007332629 wait complete. 2023-11-15T05:02:33.126 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.123+0000 7f1fa526c700 1 Processor -- start 2023-11-15T05:02:33.126 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.123+0000 7f1fa526c700 1 -- start start 2023-11-15T05:02:33.126 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.123+0000 7f1fa526c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fa00f5350 0x7f1fa0109560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:33.126 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.123+0000 7f1fa526c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa0109a50 con 0x7f1fa00f5350 2023-11-15T05:02:33.127 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.123+0000 7f1f9effd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fa00f5350 0x7f1fa0109560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:33.127 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.123+0000 7f1f9effd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fa00f5350 0x7f1fa0109560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:35658/0 (socket says 172.21.15.178:35658) 2023-11-15T05:02:33.127 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.123+0000 7f1f9effd700 1 -- 172.21.15.178:0/867320590 learned_addr learned my addr 172.21.15.178:0/867320590 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:02:33.127 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.124+0000 7f1f9effd700 1 -- 172.21.15.178:0/867320590 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f94005040 con 0x7f1fa00f5350 2023-11-15T05:02:33.127 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.124+0000 7f1f9effd700 1 --2- 172.21.15.178:0/867320590 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fa00f5350 0x7f1fa0109560 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f1f94009f80 tx=0x7f1f94009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:33.128 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.124+0000 7f1f877fe700 1 -- 172.21.15.178:0/867320590 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f940136c0 con 0x7f1fa00f5350 2023-11-15T05:02:33.128 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.124+0000 7f1f877fe700 1 -- 172.21.15.178:0/867320590 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f940095f0 con 0x7f1fa00f5350 2023-11-15T05:02:33.128 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.124+0000 7f1fa526c700 1 -- 172.21.15.178:0/867320590 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fa0109b90 con 0x7f1fa00f5350 2023-11-15T05:02:33.128 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.124+0000 7f1f877fe700 1 -- 172.21.15.178:0/867320590 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f94014420 con 0x7f1fa00f5350 2023-11-15T05:02:33.128 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.124+0000 7f1fa526c700 1 -- 172.21.15.178:0/867320590 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fa0109fc0 con 0x7f1fa00f5350 2023-11-15T05:02:33.129 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.125+0000 7f1f877fe700 1 -- 172.21.15.178:0/867320590 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1f94025060 con 0x7f1fa00f5350 2023-11-15T05:02:33.130 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.125+0000 7f1fa526c700 1 -- 172.21.15.178:0/867320590 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa0045ad0 con 0x7f1fa00f5350 2023-11-15T05:02:33.130 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.125+0000 7f1f877fe700 1 --2- 172.21.15.178:0/867320590 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1f8802ff60 0x7f1f88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:33.130 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.125+0000 7f1f877fe700 1 -- 172.21.15.178:0/867320590 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1f94040db0 con 0x7f1fa00f5350 2023-11-15T05:02:33.130 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.125+0000 7f1f9e7fc700 1 --2- 172.21.15.178:0/867320590 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1f8802ff60 0x7f1f88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:33.130 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.125+0000 7f1f9e7fc700 1 --2- 172.21.15.178:0/867320590 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1f8802ff60 0x7f1f88032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f1f900096a0 tx=0x7f1f90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:33.131 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.130+0000 7f1f877fe700 1 -- 172.21.15.178:0/867320590 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f9400f280 con 0x7f1fa00f5350 2023-11-15T05:02:33.338 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.338+0000 7f1fa526c700 1 -- 172.21.15.178:0/867320590 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1fa00fae50 con 0x7f1fa00f5350 2023-11-15T05:02:33.339 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.338+0000 7f1f877fe700 1 -- 172.21.15.178:0/867320590 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1f9402d2d0 con 0x7f1fa00f5350 2023-11-15T05:02:33.339 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:02:33.339 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:33.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.340+0000 7f1fa526c700 1 -- 172.21.15.178:0/867320590 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1f8802ff60 msgr2=0x7f1f88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:33.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.340+0000 7f1fa526c700 1 --2- 172.21.15.178:0/867320590 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1f8802ff60 0x7f1f88032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f1f900096a0 tx=0x7f1f90006b40).stop 2023-11-15T05:02:33.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.340+0000 7f1fa526c700 1 -- 172.21.15.178:0/867320590 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fa00f5350 msgr2=0x7f1fa0109560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:33.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.340+0000 7f1fa526c700 1 --2- 172.21.15.178:0/867320590 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fa00f5350 0x7f1fa0109560 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f1f94009f80 tx=0x7f1f94009b30).stop 2023-11-15T05:02:33.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.340+0000 7f1fa526c700 1 -- 172.21.15.178:0/867320590 shutdown_connections 2023-11-15T05:02:33.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.340+0000 7f1fa526c700 1 --2- 172.21.15.178:0/867320590 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1f8802ff60 0x7f1f88032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:33.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.340+0000 7f1fa526c700 1 --2- 172.21.15.178:0/867320590 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fa00f5350 0x7f1fa0109560 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:33.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.340+0000 7f1fa526c700 1 -- 172.21.15.178:0/867320590 >> 172.21.15.178:0/867320590 conn(0x7f1fa00f02e0 msgr2=0x7f1fa00f0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:33.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.340+0000 7f1fa526c700 1 -- 172.21.15.178:0/867320590 shutdown_connections 2023-11-15T05:02:33.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:33.340+0000 7f1fa526c700 1 -- 172.21.15.178:0/867320590 wait complete. 2023-11-15T05:02:33.343 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:34.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:34 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:32.376026+0000 mgr.smithi163.qesyrx (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:34.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:34 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:33.339019+0000 mon.smithi163 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.178:0/867320590' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:34.736 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:34.737 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:36.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.048+0000 7fcd3dd2d700 1 -- 172.21.15.178:0/3267624653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd38060280 msgr2=0x7fcd380f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:36.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.048+0000 7fcd3dd2d700 1 --2- 172.21.15.178:0/3267624653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd38060280 0x7fcd380f90f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fcd28009fb0 tx=0x7fcd28009b30).stop 2023-11-15T05:02:36.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.048+0000 7fcd3dd2d700 1 -- 172.21.15.178:0/3267624653 shutdown_connections 2023-11-15T05:02:36.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.048+0000 7fcd3dd2d700 1 --2- 172.21.15.178:0/3267624653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd38060280 0x7fcd380f90f0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:36.050 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.048+0000 7fcd3dd2d700 1 -- 172.21.15.178:0/3267624653 >> 172.21.15.178:0/3267624653 conn(0x7fcd380f0b70 msgr2=0x7fcd380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:36.050 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.048+0000 7fcd3dd2d700 1 -- 172.21.15.178:0/3267624653 shutdown_connections 2023-11-15T05:02:36.050 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.048+0000 7fcd3dd2d700 1 -- 172.21.15.178:0/3267624653 wait complete. 2023-11-15T05:02:36.052 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.049+0000 7fcd3dd2d700 1 Processor -- start 2023-11-15T05:02:36.052 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.049+0000 7fcd3dd2d700 1 -- start start 2023-11-15T05:02:36.052 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.049+0000 7fcd3dd2d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd38060280 0x7fcd381066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:36.053 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.049+0000 7fcd3dd2d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd38106bd0 con 0x7fcd38060280 2023-11-15T05:02:36.053 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.049+0000 7fcd377fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd38060280 0x7fcd381066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:36.053 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.049+0000 7fcd377fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd38060280 0x7fcd381066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:51794/0 (socket says 172.21.15.178:51794) 2023-11-15T05:02:36.053 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.049+0000 7fcd377fe700 1 -- 172.21.15.178:0/352239153 learned_addr learned my addr 172.21.15.178:0/352239153 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:02:36.053 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.050+0000 7fcd377fe700 1 -- 172.21.15.178:0/352239153 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd28005040 con 0x7fcd38060280 2023-11-15T05:02:36.054 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.050+0000 7fcd377fe700 1 --2- 172.21.15.178:0/352239153 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd38060280 0x7fcd381066e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fcd28013510 tx=0x7fcd2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:36.054 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.050+0000 7fcd1ffff700 1 -- 172.21.15.178:0/352239153 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd28009320 con 0x7fcd38060280 2023-11-15T05:02:36.054 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.050+0000 7fcd3dd2d700 1 -- 172.21.15.178:0/352239153 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd38106dd0 con 0x7fcd38060280 2023-11-15T05:02:36.054 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.050+0000 7fcd1ffff700 1 -- 172.21.15.178:0/352239153 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd28013f00 con 0x7fcd38060280 2023-11-15T05:02:36.055 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.050+0000 7fcd1ffff700 1 -- 172.21.15.178:0/352239153 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd28014dc0 con 0x7fcd38060280 2023-11-15T05:02:36.055 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.050+0000 7fcd3dd2d700 1 -- 172.21.15.178:0/352239153 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd38107260 con 0x7fcd38060280 2023-11-15T05:02:36.055 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.051+0000 7fcd1ffff700 1 -- 172.21.15.178:0/352239153 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcd280095f0 con 0x7fcd38060280 2023-11-15T05:02:36.055 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.051+0000 7fcd1ffff700 1 --2- 172.21.15.178:0/352239153 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcd2002ff60 0x7fcd20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:36.055 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.051+0000 7fcd3dd2d700 1 -- 172.21.15.178:0/352239153 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd38045ad0 con 0x7fcd38060280 2023-11-15T05:02:36.055 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.051+0000 7fcd1ffff700 1 -- 172.21.15.178:0/352239153 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcd28041030 con 0x7fcd38060280 2023-11-15T05:02:36.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.051+0000 7fcd36ffd700 1 --2- 172.21.15.178:0/352239153 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcd2002ff60 0x7fcd20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:36.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.052+0000 7fcd36ffd700 1 --2- 172.21.15.178:0/352239153 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcd2002ff60 0x7fcd20032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fcd2c0096a0 tx=0x7fcd2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:36.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.056+0000 7fcd1ffff700 1 -- 172.21.15.178:0/352239153 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd28014420 con 0x7fcd38060280 2023-11-15T05:02:36.260 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.259+0000 7fcd3dd2d700 1 -- 172.21.15.178:0/352239153 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd38108110 con 0x7fcd38060280 2023-11-15T05:02:36.260 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.260+0000 7fcd1ffff700 1 -- 172.21.15.178:0/352239153 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcd28010230 con 0x7fcd38060280 2023-11-15T05:02:36.261 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:02:36.261 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:36.262 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.261+0000 7fcd3dd2d700 1 -- 172.21.15.178:0/352239153 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcd2002ff60 msgr2=0x7fcd20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:36.262 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.261+0000 7fcd3dd2d700 1 --2- 172.21.15.178:0/352239153 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcd2002ff60 0x7fcd20032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fcd2c0096a0 tx=0x7fcd2c006b40).stop 2023-11-15T05:02:36.262 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.261+0000 7fcd3dd2d700 1 -- 172.21.15.178:0/352239153 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd38060280 msgr2=0x7fcd381066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:36.262 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.261+0000 7fcd3dd2d700 1 --2- 172.21.15.178:0/352239153 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd38060280 0x7fcd381066e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fcd28013510 tx=0x7fcd2800b040).stop 2023-11-15T05:02:36.263 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.261+0000 7fcd3dd2d700 1 -- 172.21.15.178:0/352239153 shutdown_connections 2023-11-15T05:02:36.263 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.261+0000 7fcd3dd2d700 1 --2- 172.21.15.178:0/352239153 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcd2002ff60 0x7fcd20032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:36.263 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.261+0000 7fcd3dd2d700 1 --2- 172.21.15.178:0/352239153 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd38060280 0x7fcd381066e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:36.263 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.261+0000 7fcd3dd2d700 1 -- 172.21.15.178:0/352239153 >> 172.21.15.178:0/352239153 conn(0x7fcd380f0b70 msgr2=0x7fcd380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:36.263 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.262+0000 7fcd3dd2d700 1 -- 172.21.15.178:0/352239153 shutdown_connections 2023-11-15T05:02:36.264 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:36.262+0000 7fcd3dd2d700 1 -- 172.21.15.178:0/352239153 wait complete. 2023-11-15T05:02:36.266 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:36.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:36 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:34.376346+0000 mgr.smithi163.qesyrx (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:37.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:37 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:36.260552+0000 mon.smithi163 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.178:0/352239153' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:37.650 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:37.650 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:38.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:38 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:36.376704+0000 mgr.smithi163.qesyrx (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:38.838 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.837+0000 7fdc93558700 1 -- 172.21.15.178:0/4047523122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc8c0f5410 msgr2=0x7fdc8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:38.838 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.837+0000 7fdc93558700 1 --2- 172.21.15.178:0/4047523122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc8c0f5410 0x7fdc8c0f5830 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fdc7c009fb0 tx=0x7fdc7c009b30).stop 2023-11-15T05:02:38.838 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.837+0000 7fdc93558700 1 -- 172.21.15.178:0/4047523122 shutdown_connections 2023-11-15T05:02:38.839 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.837+0000 7fdc93558700 1 --2- 172.21.15.178:0/4047523122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc8c0f5410 0x7fdc8c0f5830 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:38.839 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.837+0000 7fdc93558700 1 -- 172.21.15.178:0/4047523122 >> 172.21.15.178:0/4047523122 conn(0x7fdc8c0f0b70 msgr2=0x7fdc8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:38.840 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.837+0000 7fdc93558700 1 -- 172.21.15.178:0/4047523122 shutdown_connections 2023-11-15T05:02:38.840 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.837+0000 7fdc93558700 1 -- 172.21.15.178:0/4047523122 wait complete. 2023-11-15T05:02:38.844 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.838+0000 7fdc93558700 1 Processor -- start 2023-11-15T05:02:38.845 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.838+0000 7fdc93558700 1 -- start start 2023-11-15T05:02:38.845 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.838+0000 7fdc93558700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc8c0f5410 0x7fdc8c109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:38.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.838+0000 7fdc93558700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c060240 con 0x7fdc8c0f5410 2023-11-15T05:02:38.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.838+0000 7fdc912f4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc8c0f5410 0x7fdc8c109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:38.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.838+0000 7fdc912f4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc8c0f5410 0x7fdc8c109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:51814/0 (socket says 172.21.15.178:51814) 2023-11-15T05:02:38.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.838+0000 7fdc912f4700 1 -- 172.21.15.178:0/2913226826 learned_addr learned my addr 172.21.15.178:0/2913226826 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:02:38.848 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.839+0000 7fdc912f4700 1 -- 172.21.15.178:0/2913226826 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c005040 con 0x7fdc8c0f5410 2023-11-15T05:02:38.848 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.839+0000 7fdc912f4700 1 --2- 172.21.15.178:0/2913226826 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc8c0f5410 0x7fdc8c109400 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fdc7c013510 tx=0x7fdc7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:38.848 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.839+0000 7fdc81ffb700 1 -- 172.21.15.178:0/2913226826 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc7c009320 con 0x7fdc8c0f5410 2023-11-15T05:02:38.849 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.839+0000 7fdc81ffb700 1 -- 172.21.15.178:0/2913226826 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc7c013f00 con 0x7fdc8c0f5410 2023-11-15T05:02:38.849 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.839+0000 7fdc93558700 1 -- 172.21.15.178:0/2913226826 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc8c060440 con 0x7fdc8c0f5410 2023-11-15T05:02:38.849 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.839+0000 7fdc81ffb700 1 -- 172.21.15.178:0/2913226826 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc7c014d90 con 0x7fdc8c0f5410 2023-11-15T05:02:38.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.839+0000 7fdc93558700 1 -- 172.21.15.178:0/2913226826 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc8c109ab0 con 0x7fdc8c0f5410 2023-11-15T05:02:38.850 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.840+0000 7fdc81ffb700 1 -- 172.21.15.178:0/2913226826 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc7c0095f0 con 0x7fdc8c0f5410 2023-11-15T05:02:38.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.840+0000 7fdc93558700 1 -- 172.21.15.178:0/2913226826 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc8c045ad0 con 0x7fdc8c0f5410 2023-11-15T05:02:38.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.840+0000 7fdc81ffb700 1 --2- 172.21.15.178:0/2913226826 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdc7802ff60 0x7fdc78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:38.851 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.840+0000 7fdc81ffb700 1 -- 172.21.15.178:0/2913226826 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc7c0410a0 con 0x7fdc8c0f5410 2023-11-15T05:02:38.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.840+0000 7fdc90af3700 1 --2- 172.21.15.178:0/2913226826 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdc7802ff60 0x7fdc78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:38.852 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.841+0000 7fdc90af3700 1 --2- 172.21.15.178:0/2913226826 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdc7802ff60 0x7fdc78032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fdc880096a0 tx=0x7fdc88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:38.853 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:38.845+0000 7fdc81ffb700 1 -- 172.21.15.178:0/2913226826 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc7c01d7f0 con 0x7fdc8c0f5410 2023-11-15T05:02:39.052 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:39.052+0000 7fdc93558700 1 -- 172.21.15.178:0/2913226826 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc8c0fae60 con 0x7fdc8c0f5410 2023-11-15T05:02:39.053 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:39.052+0000 7fdc81ffb700 1 -- 172.21.15.178:0/2913226826 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdc7c010230 con 0x7fdc8c0f5410 2023-11-15T05:02:39.053 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:02:39.053 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:39.055 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:39.054+0000 7fdc93558700 1 -- 172.21.15.178:0/2913226826 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdc7802ff60 msgr2=0x7fdc78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:39.055 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:39.054+0000 7fdc93558700 1 --2- 172.21.15.178:0/2913226826 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdc7802ff60 0x7fdc78032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fdc880096a0 tx=0x7fdc88006b40).stop 2023-11-15T05:02:39.055 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:39.054+0000 7fdc93558700 1 -- 172.21.15.178:0/2913226826 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc8c0f5410 msgr2=0x7fdc8c109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:39.055 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:39.054+0000 7fdc93558700 1 --2- 172.21.15.178:0/2913226826 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc8c0f5410 0x7fdc8c109400 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fdc7c013510 tx=0x7fdc7c00b040).stop 2023-11-15T05:02:39.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:39.054+0000 7fdc93558700 1 -- 172.21.15.178:0/2913226826 shutdown_connections 2023-11-15T05:02:39.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:39.054+0000 7fdc93558700 1 --2- 172.21.15.178:0/2913226826 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdc7802ff60 0x7fdc78032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:39.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:39.054+0000 7fdc93558700 1 --2- 172.21.15.178:0/2913226826 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc8c0f5410 0x7fdc8c109400 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:39.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:39.054+0000 7fdc93558700 1 -- 172.21.15.178:0/2913226826 >> 172.21.15.178:0/2913226826 conn(0x7fdc8c0f0b70 msgr2=0x7fdc8c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:39.057 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:39.054+0000 7fdc93558700 1 -- 172.21.15.178:0/2913226826 shutdown_connections 2023-11-15T05:02:39.057 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:39.054+0000 7fdc93558700 1 -- 172.21.15.178:0/2913226826 wait complete. 2023-11-15T05:02:39.058 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:40.438 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:40.439 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:40.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:40 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:38.377010+0000 mgr.smithi163.qesyrx (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:40.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:40 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:39.053095+0000 mon.smithi163 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.178:0/2913226826' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:41.658 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.658+0000 7f26fd60b700 1 -- 172.21.15.178:0/1700676974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26f8060280 msgr2=0x7f26f80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:41.659 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.658+0000 7f26fd60b700 1 --2- 172.21.15.178:0/1700676974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26f8060280 0x7f26f80f90f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f26ec009fb0 tx=0x7f26ec009b30).stop 2023-11-15T05:02:41.659 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.658+0000 7f26fd60b700 1 -- 172.21.15.178:0/1700676974 shutdown_connections 2023-11-15T05:02:41.659 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.658+0000 7f26fd60b700 1 --2- 172.21.15.178:0/1700676974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26f8060280 0x7f26f80f90f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:41.659 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.658+0000 7f26fd60b700 1 -- 172.21.15.178:0/1700676974 >> 172.21.15.178:0/1700676974 conn(0x7f26f80f0b70 msgr2=0x7f26f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:41.660 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.658+0000 7f26fd60b700 1 -- 172.21.15.178:0/1700676974 shutdown_connections 2023-11-15T05:02:41.660 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.658+0000 7f26fd60b700 1 -- 172.21.15.178:0/1700676974 wait complete. 2023-11-15T05:02:41.662 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.659+0000 7f26fd60b700 1 Processor -- start 2023-11-15T05:02:41.662 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.659+0000 7f26fd60b700 1 -- start start 2023-11-15T05:02:41.662 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.659+0000 7f26fd60b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26f8060280 0x7f26f81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:41.663 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.659+0000 7f26fd60b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f8106bd0 con 0x7f26f8060280 2023-11-15T05:02:41.663 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.659+0000 7f26f6ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26f8060280 0x7f26f81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:41.663 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.659+0000 7f26f6ffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26f8060280 0x7f26f81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:51840/0 (socket says 172.21.15.178:51840) 2023-11-15T05:02:41.664 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.659+0000 7f26f6ffd700 1 -- 172.21.15.178:0/666728472 learned_addr learned my addr 172.21.15.178:0/666728472 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:02:41.664 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.659+0000 7f26f6ffd700 1 -- 172.21.15.178:0/666728472 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ec005040 con 0x7f26f8060280 2023-11-15T05:02:41.664 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.660+0000 7f26f6ffd700 1 --2- 172.21.15.178:0/666728472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26f8060280 0x7f26f81066e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f26ec013510 tx=0x7f26ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:41.664 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.660+0000 7f26df7fe700 1 -- 172.21.15.178:0/666728472 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26ec009320 con 0x7f26f8060280 2023-11-15T05:02:41.664 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.660+0000 7f26fd60b700 1 -- 172.21.15.178:0/666728472 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f8106dd0 con 0x7f26f8060280 2023-11-15T05:02:41.664 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.660+0000 7f26df7fe700 1 -- 172.21.15.178:0/666728472 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f26ec013f00 con 0x7f26f8060280 2023-11-15T05:02:41.665 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.660+0000 7f26df7fe700 1 -- 172.21.15.178:0/666728472 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26ec014dc0 con 0x7f26f8060280 2023-11-15T05:02:41.665 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.660+0000 7f26fd60b700 1 -- 172.21.15.178:0/666728472 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f8107260 con 0x7f26f8060280 2023-11-15T05:02:41.665 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.660+0000 7f26df7fe700 1 -- 172.21.15.178:0/666728472 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26ec01d3d0 con 0x7f26f8060280 2023-11-15T05:02:41.665 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.661+0000 7f26df7fe700 1 --2- 172.21.15.178:0/666728472 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f26e002ff60 0x7f26e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:41.665 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.661+0000 7f26df7fe700 1 -- 172.21.15.178:0/666728472 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f26ec041010 con 0x7f26f8060280 2023-11-15T05:02:41.666 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.661+0000 7f26fd60b700 1 -- 172.21.15.178:0/666728472 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f8045ad0 con 0x7f26f8060280 2023-11-15T05:02:41.666 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.661+0000 7f26f67fc700 1 --2- 172.21.15.178:0/666728472 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f26e002ff60 0x7f26e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:41.666 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.661+0000 7f26f67fc700 1 --2- 172.21.15.178:0/666728472 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f26e002ff60 0x7f26e0032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f26e80096a0 tx=0x7f26e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:41.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.666+0000 7f26df7fe700 1 -- 172.21.15.178:0/666728472 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f26ec012900 con 0x7f26f8060280 2023-11-15T05:02:41.869 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.869+0000 7f26fd60b700 1 -- 172.21.15.178:0/666728472 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26f8107f10 con 0x7f26f8060280 2023-11-15T05:02:41.870 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.869+0000 7f26df7fe700 1 -- 172.21.15.178:0/666728472 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f26ec02b020 con 0x7f26f8060280 2023-11-15T05:02:41.870 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:02:41.870 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:41.871 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.871+0000 7f26fd60b700 1 -- 172.21.15.178:0/666728472 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f26e002ff60 msgr2=0x7f26e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:41.872 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.871+0000 7f26fd60b700 1 --2- 172.21.15.178:0/666728472 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f26e002ff60 0x7f26e0032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f26e80096a0 tx=0x7f26e8006b40).stop 2023-11-15T05:02:41.872 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.871+0000 7f26fd60b700 1 -- 172.21.15.178:0/666728472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26f8060280 msgr2=0x7f26f81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:41.872 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.871+0000 7f26fd60b700 1 --2- 172.21.15.178:0/666728472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26f8060280 0x7f26f81066e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f26ec013510 tx=0x7f26ec00b040).stop 2023-11-15T05:02:41.872 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.871+0000 7f26fd60b700 1 -- 172.21.15.178:0/666728472 shutdown_connections 2023-11-15T05:02:41.872 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.871+0000 7f26fd60b700 1 --2- 172.21.15.178:0/666728472 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f26e002ff60 0x7f26e0032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:41.873 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.871+0000 7f26fd60b700 1 --2- 172.21.15.178:0/666728472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26f8060280 0x7f26f81066e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:41.873 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.871+0000 7f26fd60b700 1 -- 172.21.15.178:0/666728472 >> 172.21.15.178:0/666728472 conn(0x7f26f80f0b70 msgr2=0x7f26f80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:41.873 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.871+0000 7f26fd60b700 1 -- 172.21.15.178:0/666728472 shutdown_connections 2023-11-15T05:02:41.873 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:41.871+0000 7f26fd60b700 1 -- 172.21.15.178:0/666728472 wait complete. 2023-11-15T05:02:41.874 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:42.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:42 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:40.377303+0000 mgr.smithi163.qesyrx (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:42.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:42 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:41.869993+0000 mon.smithi163 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.178:0/666728472' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:43.307 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:43.307 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:44.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:44 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:42.377648+0000 mgr.smithi163.qesyrx (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:44.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.481+0000 7f41fc059700 1 -- 172.21.15.178:0/2408101922 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41f40f6430 msgr2=0x7f41f40f6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:44.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.481+0000 7f41fc059700 1 --2- 172.21.15.178:0/2408101922 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41f40f6430 0x7f41f40f6850 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f41e8009fb0 tx=0x7f41e8009b30).stop 2023-11-15T05:02:44.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.482+0000 7f41fc059700 1 -- 172.21.15.178:0/2408101922 shutdown_connections 2023-11-15T05:02:44.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.482+0000 7f41fc059700 1 --2- 172.21.15.178:0/2408101922 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41f40f6430 0x7f41f40f6850 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:44.483 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.482+0000 7f41fc059700 1 -- 172.21.15.178:0/2408101922 >> 172.21.15.178:0/2408101922 conn(0x7f41f40f1b90 msgr2=0x7f41f40f3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:44.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.482+0000 7f41fc059700 1 -- 172.21.15.178:0/2408101922 shutdown_connections 2023-11-15T05:02:44.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.482+0000 7f41fc059700 1 -- 172.21.15.178:0/2408101922 wait complete. 2023-11-15T05:02:44.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.482+0000 7f41fc059700 1 Processor -- start 2023-11-15T05:02:44.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.482+0000 7f41fc059700 1 -- start start 2023-11-15T05:02:44.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.482+0000 7f41fc059700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41f40f6430 0x7f41f406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:44.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.482+0000 7f41fc059700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f4069e20 con 0x7f41f40f6430 2023-11-15T05:02:44.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.482+0000 7f41f9df5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41f40f6430 0x7f41f406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:44.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.483+0000 7f41f9df5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41f40f6430 0x7f41f406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:53578/0 (socket says 172.21.15.178:53578) 2023-11-15T05:02:44.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.483+0000 7f41f9df5700 1 -- 172.21.15.178:0/2352000443 learned_addr learned my addr 172.21.15.178:0/2352000443 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:02:44.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.483+0000 7f41f9df5700 1 -- 172.21.15.178:0/2352000443 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e8005040 con 0x7f41f40f6430 2023-11-15T05:02:44.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.483+0000 7f41f9df5700 1 --2- 172.21.15.178:0/2352000443 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41f40f6430 0x7f41f406b9a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f41e8013510 tx=0x7f41e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:44.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.483+0000 7f41e67fc700 1 -- 172.21.15.178:0/2352000443 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41e8009320 con 0x7f41f40f6430 2023-11-15T05:02:44.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.483+0000 7f41fc059700 1 -- 172.21.15.178:0/2352000443 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f4069fc0 con 0x7f41f40f6430 2023-11-15T05:02:44.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.483+0000 7f41e67fc700 1 -- 172.21.15.178:0/2352000443 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f41e8013f00 con 0x7f41f40f6430 2023-11-15T05:02:44.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.483+0000 7f41e67fc700 1 -- 172.21.15.178:0/2352000443 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41e8014d90 con 0x7f41f40f6430 2023-11-15T05:02:44.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.483+0000 7f41fc059700 1 -- 172.21.15.178:0/2352000443 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f406a450 con 0x7f41f40f6430 2023-11-15T05:02:44.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.484+0000 7f41e67fc700 1 -- 172.21.15.178:0/2352000443 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f41e80095f0 con 0x7f41f40f6430 2023-11-15T05:02:44.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.484+0000 7f41fc059700 1 -- 172.21.15.178:0/2352000443 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f4045ad0 con 0x7f41f40f6430 2023-11-15T05:02:44.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.484+0000 7f41e67fc700 1 --2- 172.21.15.178:0/2352000443 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f41e002ffb0 0x7f41e0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:44.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.484+0000 7f41e67fc700 1 -- 172.21.15.178:0/2352000443 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f41e8040fe0 con 0x7f41f40f6430 2023-11-15T05:02:44.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.485+0000 7f41f95f4700 1 --2- 172.21.15.178:0/2352000443 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f41e002ffb0 0x7f41e0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:44.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.485+0000 7f41f95f4700 1 --2- 172.21.15.178:0/2352000443 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f41e002ffb0 0x7f41e0032470 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f41f00096a0 tx=0x7f41f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:44.492 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.490+0000 7f41e67fc700 1 -- 172.21.15.178:0/2352000443 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f41e806d2b0 con 0x7f41f40f6430 2023-11-15T05:02:44.695 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.695+0000 7f41fc059700 1 -- 172.21.15.178:0/2352000443 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f41f405ea50 con 0x7f41f40f6430 2023-11-15T05:02:44.696 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.695+0000 7f41e67fc700 1 -- 172.21.15.178:0/2352000443 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f41e8014420 con 0x7f41f40f6430 2023-11-15T05:02:44.696 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:02:44.696 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:44.697 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.697+0000 7f41fc059700 1 -- 172.21.15.178:0/2352000443 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f41e002ffb0 msgr2=0x7f41e0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:44.698 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.697+0000 7f41fc059700 1 --2- 172.21.15.178:0/2352000443 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f41e002ffb0 0x7f41e0032470 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f41f00096a0 tx=0x7f41f0006b40).stop 2023-11-15T05:02:44.698 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.697+0000 7f41fc059700 1 -- 172.21.15.178:0/2352000443 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41f40f6430 msgr2=0x7f41f406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:44.698 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.697+0000 7f41fc059700 1 --2- 172.21.15.178:0/2352000443 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41f40f6430 0x7f41f406b9a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f41e8013510 tx=0x7f41e800b040).stop 2023-11-15T05:02:44.698 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.697+0000 7f41fc059700 1 -- 172.21.15.178:0/2352000443 shutdown_connections 2023-11-15T05:02:44.698 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.697+0000 7f41fc059700 1 --2- 172.21.15.178:0/2352000443 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f41e002ffb0 0x7f41e0032470 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:44.699 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.697+0000 7f41fc059700 1 --2- 172.21.15.178:0/2352000443 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41f40f6430 0x7f41f406b9a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:44.699 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.697+0000 7f41fc059700 1 -- 172.21.15.178:0/2352000443 >> 172.21.15.178:0/2352000443 conn(0x7f41f40f1b90 msgr2=0x7f41f40f2630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:44.699 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.697+0000 7f41fc059700 1 -- 172.21.15.178:0/2352000443 shutdown_connections 2023-11-15T05:02:44.699 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:44.697+0000 7f41fc059700 1 -- 172.21.15.178:0/2352000443 wait complete. 2023-11-15T05:02:44.700 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:45.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:45 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:44.696281+0000 mon.smithi163 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.178:0/2352000443' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:46.091 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:46.092 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:46.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:46 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:44.377981+0000 mgr.smithi163.qesyrx (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:47.300 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.298+0000 7f8af5283700 1 -- 172.21.15.178:0/1859826556 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8af006b960 msgr2=0x7f8af0069e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:47.300 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.298+0000 7f8af5283700 1 --2- 172.21.15.178:0/1859826556 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8af006b960 0x7f8af0069e60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f8ae0009fb0 tx=0x7f8ae0009b30).stop 2023-11-15T05:02:47.300 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.298+0000 7f8af5283700 1 -- 172.21.15.178:0/1859826556 shutdown_connections 2023-11-15T05:02:47.301 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.298+0000 7f8af5283700 1 --2- 172.21.15.178:0/1859826556 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8af006b960 0x7f8af0069e60 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:47.301 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.298+0000 7f8af5283700 1 -- 172.21.15.178:0/1859826556 >> 172.21.15.178:0/1859826556 conn(0x7f8af00f1c70 msgr2=0x7f8af00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:47.301 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.298+0000 7f8af5283700 1 -- 172.21.15.178:0/1859826556 shutdown_connections 2023-11-15T05:02:47.301 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.298+0000 7f8af5283700 1 -- 172.21.15.178:0/1859826556 wait complete. 2023-11-15T05:02:47.301 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.299+0000 7f8af5283700 1 Processor -- start 2023-11-15T05:02:47.302 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.299+0000 7f8af5283700 1 -- start start 2023-11-15T05:02:47.302 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.299+0000 7f8af5283700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8af006b960 0x7f8af01088e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:47.302 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.299+0000 7f8af5283700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af0108dd0 con 0x7f8af006b960 2023-11-15T05:02:47.302 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.299+0000 7f8aeeffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8af006b960 0x7f8af01088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:47.302 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.299+0000 7f8aeeffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8af006b960 0x7f8af01088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:53604/0 (socket says 172.21.15.178:53604) 2023-11-15T05:02:47.303 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.299+0000 7f8aeeffd700 1 -- 172.21.15.178:0/3783008197 learned_addr learned my addr 172.21.15.178:0/3783008197 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:02:47.303 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.299+0000 7f8aeeffd700 1 -- 172.21.15.178:0/3783008197 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae0005040 con 0x7f8af006b960 2023-11-15T05:02:47.303 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.299+0000 7f8aeeffd700 1 --2- 172.21.15.178:0/3783008197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8af006b960 0x7f8af01088e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f8ae0013510 tx=0x7f8ae000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:47.303 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.300+0000 7f8ad77fe700 1 -- 172.21.15.178:0/3783008197 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ae0009320 con 0x7f8af006b960 2023-11-15T05:02:47.303 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.300+0000 7f8ad77fe700 1 -- 172.21.15.178:0/3783008197 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ae0013f00 con 0x7f8af006b960 2023-11-15T05:02:47.304 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.300+0000 7f8af5283700 1 -- 172.21.15.178:0/3783008197 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af0108fd0 con 0x7f8af006b960 2023-11-15T05:02:47.304 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.300+0000 7f8ad77fe700 1 -- 172.21.15.178:0/3783008197 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ae0014dc0 con 0x7f8af006b960 2023-11-15T05:02:47.304 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.300+0000 7f8af5283700 1 -- 172.21.15.178:0/3783008197 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af0109460 con 0x7f8af006b960 2023-11-15T05:02:47.305 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.300+0000 7f8ad77fe700 1 -- 172.21.15.178:0/3783008197 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ae001d3d0 con 0x7f8af006b960 2023-11-15T05:02:47.305 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.301+0000 7f8ad77fe700 1 --2- 172.21.15.178:0/3783008197 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8ad802ff60 0x7f8ad8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:47.305 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.301+0000 7f8af5283700 1 -- 172.21.15.178:0/3783008197 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af0045ad0 con 0x7f8af006b960 2023-11-15T05:02:47.305 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.301+0000 7f8ad77fe700 1 -- 172.21.15.178:0/3783008197 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ae0040fe0 con 0x7f8af006b960 2023-11-15T05:02:47.306 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.301+0000 7f8aee7fc700 1 --2- 172.21.15.178:0/3783008197 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8ad802ff60 0x7f8ad8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:47.306 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.301+0000 7f8aee7fc700 1 --2- 172.21.15.178:0/3783008197 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8ad802ff60 0x7f8ad8032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f8ae40096a0 tx=0x7f8ae4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:47.306 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.306+0000 7f8ad77fe700 1 -- 172.21.15.178:0/3783008197 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ae006d2b0 con 0x7f8af006b960 2023-11-15T05:02:47.510 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.510+0000 7f8af5283700 1 -- 172.21.15.178:0/3783008197 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8af010a310 con 0x7f8af006b960 2023-11-15T05:02:47.511 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.511+0000 7f8ad77fe700 1 -- 172.21.15.178:0/3783008197 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8ae001db80 con 0x7f8af006b960 2023-11-15T05:02:47.512 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:02:47.512 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:47.513 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.513+0000 7f8af5283700 1 -- 172.21.15.178:0/3783008197 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8ad802ff60 msgr2=0x7f8ad8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:47.513 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.513+0000 7f8af5283700 1 --2- 172.21.15.178:0/3783008197 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8ad802ff60 0x7f8ad8032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f8ae40096a0 tx=0x7f8ae4006b40).stop 2023-11-15T05:02:47.513 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.513+0000 7f8af5283700 1 -- 172.21.15.178:0/3783008197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8af006b960 msgr2=0x7f8af01088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:47.514 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.513+0000 7f8af5283700 1 --2- 172.21.15.178:0/3783008197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8af006b960 0x7f8af01088e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f8ae0013510 tx=0x7f8ae000b040).stop 2023-11-15T05:02:47.514 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.513+0000 7f8af5283700 1 -- 172.21.15.178:0/3783008197 shutdown_connections 2023-11-15T05:02:47.514 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.513+0000 7f8af5283700 1 --2- 172.21.15.178:0/3783008197 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8ad802ff60 0x7f8ad8032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:47.514 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.513+0000 7f8af5283700 1 --2- 172.21.15.178:0/3783008197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8af006b960 0x7f8af01088e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:47.514 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.513+0000 7f8af5283700 1 -- 172.21.15.178:0/3783008197 >> 172.21.15.178:0/3783008197 conn(0x7f8af00f1c70 msgr2=0x7f8af00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:47.514 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.513+0000 7f8af5283700 1 -- 172.21.15.178:0/3783008197 shutdown_connections 2023-11-15T05:02:47.515 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:47.513+0000 7f8af5283700 1 -- 172.21.15.178:0/3783008197 wait complete. 2023-11-15T05:02:47.515 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:48.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:48 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:46.378303+0000 mgr.smithi163.qesyrx (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:48.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:48 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:47.511816+0000 mon.smithi163 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.178:0/3783008197' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:48.909 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:48.910 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:50.137 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.136+0000 7f5d2a5f9700 1 -- 172.21.15.178:0/3505436884 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d240f62c0 msgr2=0x7f5d240f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:50.137 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.136+0000 7f5d2a5f9700 1 --2- 172.21.15.178:0/3505436884 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d240f62c0 0x7f5d240f66e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f5d20009fb0 tx=0x7f5d20009b30).stop 2023-11-15T05:02:50.138 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.136+0000 7f5d2a5f9700 1 -- 172.21.15.178:0/3505436884 shutdown_connections 2023-11-15T05:02:50.138 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.136+0000 7f5d2a5f9700 1 --2- 172.21.15.178:0/3505436884 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d240f62c0 0x7f5d240f66e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:50.138 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.136+0000 7f5d2a5f9700 1 -- 172.21.15.178:0/3505436884 >> 172.21.15.178:0/3505436884 conn(0x7f5d240f1ac0 msgr2=0x7f5d240f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:50.138 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.136+0000 7f5d2a5f9700 1 -- 172.21.15.178:0/3505436884 shutdown_connections 2023-11-15T05:02:50.138 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.136+0000 7f5d2a5f9700 1 -- 172.21.15.178:0/3505436884 wait complete. 2023-11-15T05:02:50.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.136+0000 7f5d2a5f9700 1 Processor -- start 2023-11-15T05:02:50.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.136+0000 7f5d2a5f9700 1 -- start start 2023-11-15T05:02:50.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.136+0000 7f5d2a5f9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d240f62c0 0x7f5d24106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:50.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.136+0000 7f5d2a5f9700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d24106a10 con 0x7f5d240f62c0 2023-11-15T05:02:50.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.137+0000 7f5d295f7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d240f62c0 0x7f5d24106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:50.139 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.137+0000 7f5d295f7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d240f62c0 0x7f5d24106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:53626/0 (socket says 172.21.15.178:53626) 2023-11-15T05:02:50.140 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.137+0000 7f5d295f7700 1 -- 172.21.15.178:0/1268516040 learned_addr learned my addr 172.21.15.178:0/1268516040 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:02:50.140 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.137+0000 7f5d295f7700 1 -- 172.21.15.178:0/1268516040 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d20005040 con 0x7f5d240f62c0 2023-11-15T05:02:50.140 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.137+0000 7f5d295f7700 1 --2- 172.21.15.178:0/1268516040 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d240f62c0 0x7f5d24106520 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f5d20013510 tx=0x7f5d2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:50.141 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.137+0000 7f5d19ffb700 1 -- 172.21.15.178:0/1268516040 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d20009320 con 0x7f5d240f62c0 2023-11-15T05:02:50.142 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.137+0000 7f5d2a5f9700 1 -- 172.21.15.178:0/1268516040 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d24106c10 con 0x7f5d240f62c0 2023-11-15T05:02:50.142 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.137+0000 7f5d19ffb700 1 -- 172.21.15.178:0/1268516040 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d20013f00 con 0x7f5d240f62c0 2023-11-15T05:02:50.142 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.137+0000 7f5d19ffb700 1 -- 172.21.15.178:0/1268516040 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d20014dc0 con 0x7f5d240f62c0 2023-11-15T05:02:50.143 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.137+0000 7f5d2a5f9700 1 -- 172.21.15.178:0/1268516040 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d241070a0 con 0x7f5d240f62c0 2023-11-15T05:02:50.143 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.138+0000 7f5d19ffb700 1 -- 172.21.15.178:0/1268516040 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d200095f0 con 0x7f5d240f62c0 2023-11-15T05:02:50.143 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.138+0000 7f5d2a5f9700 1 -- 172.21.15.178:0/1268516040 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d24045ad0 con 0x7f5d240f62c0 2023-11-15T05:02:50.143 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.139+0000 7f5d19ffb700 1 --2- 172.21.15.178:0/1268516040 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5d1002ff60 0x7f5d10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:50.144 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.139+0000 7f5d19ffb700 1 -- 172.21.15.178:0/1268516040 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d20041040 con 0x7f5d240f62c0 2023-11-15T05:02:50.144 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.139+0000 7f5d28df6700 1 --2- 172.21.15.178:0/1268516040 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5d1002ff60 0x7f5d10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:50.144 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.139+0000 7f5d28df6700 1 --2- 172.21.15.178:0/1268516040 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5d1002ff60 0x7f5d10032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f5d140096a0 tx=0x7f5d14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:50.144 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.143+0000 7f5d19ffb700 1 -- 172.21.15.178:0/1268516040 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d20014420 con 0x7f5d240f62c0 2023-11-15T05:02:50.351 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.351+0000 7f5d2a5f9700 1 -- 172.21.15.178:0/1268516040 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d24107f50 con 0x7f5d240f62c0 2023-11-15T05:02:50.352 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.351+0000 7f5d19ffb700 1 -- 172.21.15.178:0/1268516040 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5d20010230 con 0x7f5d240f62c0 2023-11-15T05:02:50.352 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:02:50.352 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:50.354 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.353+0000 7f5d2a5f9700 1 -- 172.21.15.178:0/1268516040 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5d1002ff60 msgr2=0x7f5d10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:50.354 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.353+0000 7f5d2a5f9700 1 --2- 172.21.15.178:0/1268516040 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5d1002ff60 0x7f5d10032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f5d140096a0 tx=0x7f5d14006b40).stop 2023-11-15T05:02:50.354 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.353+0000 7f5d2a5f9700 1 -- 172.21.15.178:0/1268516040 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d240f62c0 msgr2=0x7f5d24106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:50.354 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.353+0000 7f5d2a5f9700 1 --2- 172.21.15.178:0/1268516040 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d240f62c0 0x7f5d24106520 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f5d20013510 tx=0x7f5d2000b040).stop 2023-11-15T05:02:50.355 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.353+0000 7f5d2a5f9700 1 -- 172.21.15.178:0/1268516040 shutdown_connections 2023-11-15T05:02:50.355 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.353+0000 7f5d2a5f9700 1 --2- 172.21.15.178:0/1268516040 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5d1002ff60 0x7f5d10032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:50.355 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.353+0000 7f5d2a5f9700 1 --2- 172.21.15.178:0/1268516040 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5d240f62c0 0x7f5d24106520 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:50.355 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.353+0000 7f5d2a5f9700 1 -- 172.21.15.178:0/1268516040 >> 172.21.15.178:0/1268516040 conn(0x7f5d240f1ac0 msgr2=0x7f5d240f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:50.355 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.353+0000 7f5d2a5f9700 1 -- 172.21.15.178:0/1268516040 shutdown_connections 2023-11-15T05:02:50.356 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:50.353+0000 7f5d2a5f9700 1 -- 172.21.15.178:0/1268516040 wait complete. 2023-11-15T05:02:50.356 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:50.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:50 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:48.378571+0000 mgr.smithi163.qesyrx (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:51.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:51 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:50.352120+0000 mon.smithi163 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.178:0/1268516040' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:51.755 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:51.755 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:52.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:52 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:50.378887+0000 mgr.smithi163.qesyrx (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:52.983 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.982+0000 7f523c8ba700 1 -- 172.21.15.178:0/843044533 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52380f5410 msgr2=0x7f52380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:52.984 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.982+0000 7f523c8ba700 1 --2- 172.21.15.178:0/843044533 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52380f5410 0x7f52380f5830 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f522c009fb0 tx=0x7f522c009b30).stop 2023-11-15T05:02:52.984 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.982+0000 7f523c8ba700 1 -- 172.21.15.178:0/843044533 shutdown_connections 2023-11-15T05:02:52.984 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.982+0000 7f523c8ba700 1 --2- 172.21.15.178:0/843044533 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52380f5410 0x7f52380f5830 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:52.984 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.982+0000 7f523c8ba700 1 -- 172.21.15.178:0/843044533 >> 172.21.15.178:0/843044533 conn(0x7f52380f0b70 msgr2=0x7f52380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:52.985 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.982+0000 7f523c8ba700 1 -- 172.21.15.178:0/843044533 shutdown_connections 2023-11-15T05:02:52.985 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.982+0000 7f523c8ba700 1 -- 172.21.15.178:0/843044533 wait complete. 2023-11-15T05:02:52.985 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.983+0000 7f523c8ba700 1 Processor -- start 2023-11-15T05:02:52.985 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.983+0000 7f523c8ba700 1 -- start start 2023-11-15T05:02:52.986 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.983+0000 7f523c8ba700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52380f5410 0x7f5238106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:52.986 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.983+0000 7f523c8ba700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5238106c00 con 0x7f52380f5410 2023-11-15T05:02:52.986 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.983+0000 7f523659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52380f5410 0x7f5238106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:52.986 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.983+0000 7f523659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52380f5410 0x7f5238106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:53646/0 (socket says 172.21.15.178:53646) 2023-11-15T05:02:52.987 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.983+0000 7f523659c700 1 -- 172.21.15.178:0/2489700492 learned_addr learned my addr 172.21.15.178:0/2489700492 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:02:52.987 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.983+0000 7f523659c700 1 -- 172.21.15.178:0/2489700492 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522c005040 con 0x7f52380f5410 2023-11-15T05:02:52.987 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.983+0000 7f523659c700 1 --2- 172.21.15.178:0/2489700492 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52380f5410 0x7f5238106710 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f522c013510 tx=0x7f522c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:52.988 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.984+0000 7f5226ffd700 1 -- 172.21.15.178:0/2489700492 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f522c009320 con 0x7f52380f5410 2023-11-15T05:02:52.989 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.984+0000 7f523c8ba700 1 -- 172.21.15.178:0/2489700492 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5238106e00 con 0x7f52380f5410 2023-11-15T05:02:52.989 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.984+0000 7f5226ffd700 1 -- 172.21.15.178:0/2489700492 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f522c013f00 con 0x7f52380f5410 2023-11-15T05:02:52.989 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.984+0000 7f5226ffd700 1 -- 172.21.15.178:0/2489700492 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f522c014d90 con 0x7f52380f5410 2023-11-15T05:02:52.989 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.984+0000 7f523c8ba700 1 -- 172.21.15.178:0/2489700492 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5238107290 con 0x7f52380f5410 2023-11-15T05:02:52.989 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.984+0000 7f5226ffd700 1 -- 172.21.15.178:0/2489700492 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f522c0263a0 con 0x7f52380f5410 2023-11-15T05:02:52.990 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.985+0000 7f5226ffd700 1 --2- 172.21.15.178:0/2489700492 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f522002ff60 0x7f5220032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:52.990 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.985+0000 7f5226ffd700 1 -- 172.21.15.178:0/2489700492 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f522c041030 con 0x7f52380f5410 2023-11-15T05:02:52.990 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.985+0000 7f523c8ba700 1 -- 172.21.15.178:0/2489700492 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5238045ad0 con 0x7f52380f5410 2023-11-15T05:02:52.990 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.985+0000 7f5235d9b700 1 --2- 172.21.15.178:0/2489700492 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f522002ff60 0x7f5220032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:52.990 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.985+0000 7f5235d9b700 1 --2- 172.21.15.178:0/2489700492 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f522002ff60 0x7f5220032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f52280096a0 tx=0x7f5228006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:52.991 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:52.990+0000 7f5226ffd700 1 -- 172.21.15.178:0/2489700492 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f522c012920 con 0x7f52380f5410 2023-11-15T05:02:53.193 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:53.193+0000 7f523c8ba700 1 -- 172.21.15.178:0/2489700492 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f52380fae60 con 0x7f52380f5410 2023-11-15T05:02:53.194 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:53.193+0000 7f5226ffd700 1 -- 172.21.15.178:0/2489700492 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f522c02b020 con 0x7f52380f5410 2023-11-15T05:02:53.194 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:02:53.194 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:53.195 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:53.195+0000 7f523c8ba700 1 -- 172.21.15.178:0/2489700492 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f522002ff60 msgr2=0x7f5220032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:53.195 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:53.195+0000 7f523c8ba700 1 --2- 172.21.15.178:0/2489700492 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f522002ff60 0x7f5220032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f52280096a0 tx=0x7f5228006b40).stop 2023-11-15T05:02:53.196 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:53.195+0000 7f523c8ba700 1 -- 172.21.15.178:0/2489700492 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52380f5410 msgr2=0x7f5238106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:53.196 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:53.195+0000 7f523c8ba700 1 --2- 172.21.15.178:0/2489700492 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52380f5410 0x7f5238106710 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f522c013510 tx=0x7f522c00b040).stop 2023-11-15T05:02:53.196 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:53.195+0000 7f523c8ba700 1 -- 172.21.15.178:0/2489700492 shutdown_connections 2023-11-15T05:02:53.196 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:53.195+0000 7f523c8ba700 1 --2- 172.21.15.178:0/2489700492 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f522002ff60 0x7f5220032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:53.196 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:53.195+0000 7f523c8ba700 1 --2- 172.21.15.178:0/2489700492 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52380f5410 0x7f5238106710 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:53.197 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:53.195+0000 7f523c8ba700 1 -- 172.21.15.178:0/2489700492 >> 172.21.15.178:0/2489700492 conn(0x7f52380f0b70 msgr2=0x7f523805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:53.197 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:53.195+0000 7f523c8ba700 1 -- 172.21.15.178:0/2489700492 shutdown_connections 2023-11-15T05:02:53.197 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:53.195+0000 7f523c8ba700 1 -- 172.21.15.178:0/2489700492 wait complete. 2023-11-15T05:02:53.197 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:54.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:54 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:52.379268+0000 mgr.smithi163.qesyrx (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:54.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:54 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:53.194115+0000 mon.smithi163 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.178:0/2489700492' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:54.579 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:54.579 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:55.730 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.729+0000 7ff2ffed3700 1 -- 172.21.15.178:0/359771063 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2f80f6510 msgr2=0x7ff2f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:55.730 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.729+0000 7ff2ffed3700 1 --2- 172.21.15.178:0/359771063 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2f80f6510 0x7ff2f80f6930 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7ff2f4009fb0 tx=0x7ff2f4009b30).stop 2023-11-15T05:02:55.731 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.729+0000 7ff2ffed3700 1 -- 172.21.15.178:0/359771063 shutdown_connections 2023-11-15T05:02:55.731 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.729+0000 7ff2ffed3700 1 --2- 172.21.15.178:0/359771063 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2f80f6510 0x7ff2f80f6930 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:55.731 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.729+0000 7ff2ffed3700 1 -- 172.21.15.178:0/359771063 >> 172.21.15.178:0/359771063 conn(0x7ff2f80f1c70 msgr2=0x7ff2f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:55.731 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.729+0000 7ff2ffed3700 1 -- 172.21.15.178:0/359771063 shutdown_connections 2023-11-15T05:02:55.732 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.729+0000 7ff2ffed3700 1 -- 172.21.15.178:0/359771063 wait complete. 2023-11-15T05:02:55.732 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.729+0000 7ff2ffed3700 1 Processor -- start 2023-11-15T05:02:55.732 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.729+0000 7ff2ffed3700 1 -- start start 2023-11-15T05:02:55.732 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.729+0000 7ff2ffed3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2f80f6510 0x7ff2f8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:55.732 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.730+0000 7ff2ffed3700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f8106c20 con 0x7ff2f80f6510 2023-11-15T05:02:55.733 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.730+0000 7ff2fdc6f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2f80f6510 0x7ff2f8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:55.733 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.730+0000 7ff2fdc6f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2f80f6510 0x7ff2f8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54414/0 (socket says 172.21.15.178:54414) 2023-11-15T05:02:55.733 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.730+0000 7ff2fdc6f700 1 -- 172.21.15.178:0/946895688 learned_addr learned my addr 172.21.15.178:0/946895688 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:02:55.733 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.730+0000 7ff2fdc6f700 1 -- 172.21.15.178:0/946895688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f4005040 con 0x7ff2f80f6510 2023-11-15T05:02:55.735 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.730+0000 7ff2fdc6f700 1 --2- 172.21.15.178:0/946895688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2f80f6510 0x7ff2f8106730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7ff2f4014000 tx=0x7ff2f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:55.735 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.730+0000 7ff2ee7fc700 1 -- 172.21.15.178:0/946895688 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2f40095f0 con 0x7ff2f80f6510 2023-11-15T05:02:55.735 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.730+0000 7ff2ffed3700 1 -- 172.21.15.178:0/946895688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f8106e20 con 0x7ff2f80f6510 2023-11-15T05:02:55.735 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.730+0000 7ff2ee7fc700 1 -- 172.21.15.178:0/946895688 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2f4007e70 con 0x7ff2f80f6510 2023-11-15T05:02:55.735 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.730+0000 7ff2ee7fc700 1 -- 172.21.15.178:0/946895688 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2f4014ea0 con 0x7ff2f80f6510 2023-11-15T05:02:55.736 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.730+0000 7ff2ffed3700 1 -- 172.21.15.178:0/946895688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f81072b0 con 0x7ff2f80f6510 2023-11-15T05:02:55.736 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.731+0000 7ff2ee7fc700 1 -- 172.21.15.178:0/946895688 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2f4025070 con 0x7ff2f80f6510 2023-11-15T05:02:55.736 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.731+0000 7ff2ffed3700 1 -- 172.21.15.178:0/946895688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f8045ad0 con 0x7ff2f80f6510 2023-11-15T05:02:55.736 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.731+0000 7ff2ee7fc700 1 --2- 172.21.15.178:0/946895688 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff2e402ffb0 0x7ff2e4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:55.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.731+0000 7ff2ee7fc700 1 -- 172.21.15.178:0/946895688 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff2f4040930 con 0x7ff2f80f6510 2023-11-15T05:02:55.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.732+0000 7ff2fd46e700 1 --2- 172.21.15.178:0/946895688 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff2e402ffb0 0x7ff2e4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:55.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.732+0000 7ff2fd46e700 1 --2- 172.21.15.178:0/946895688 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff2e402ffb0 0x7ff2e4032470 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7ff2e80096a0 tx=0x7ff2e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:55.738 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.736+0000 7ff2ee7fc700 1 -- 172.21.15.178:0/946895688 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2f40079c0 con 0x7ff2f80f6510 2023-11-15T05:02:55.943 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.941+0000 7ff2ffed3700 1 -- 172.21.15.178:0/946895688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2f80fbf60 con 0x7ff2f80f6510 2023-11-15T05:02:55.944 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.942+0000 7ff2ee7fc700 1 -- 172.21.15.178:0/946895688 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff2f400f300 con 0x7ff2f80f6510 2023-11-15T05:02:55.944 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:02:55.945 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:55.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.943+0000 7ff2ffed3700 1 -- 172.21.15.178:0/946895688 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff2e402ffb0 msgr2=0x7ff2e4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:55.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.943+0000 7ff2ffed3700 1 --2- 172.21.15.178:0/946895688 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff2e402ffb0 0x7ff2e4032470 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7ff2e80096a0 tx=0x7ff2e8006b40).stop 2023-11-15T05:02:55.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.943+0000 7ff2ffed3700 1 -- 172.21.15.178:0/946895688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2f80f6510 msgr2=0x7ff2f8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:55.946 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.943+0000 7ff2ffed3700 1 --2- 172.21.15.178:0/946895688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2f80f6510 0x7ff2f8106730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7ff2f4014000 tx=0x7ff2f400b040).stop 2023-11-15T05:02:55.947 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.943+0000 7ff2ffed3700 1 -- 172.21.15.178:0/946895688 shutdown_connections 2023-11-15T05:02:55.947 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.943+0000 7ff2ffed3700 1 --2- 172.21.15.178:0/946895688 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff2e402ffb0 0x7ff2e4032470 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:55.948 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.943+0000 7ff2ffed3700 1 --2- 172.21.15.178:0/946895688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff2f80f6510 0x7ff2f8106730 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:55.949 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.943+0000 7ff2ffed3700 1 -- 172.21.15.178:0/946895688 >> 172.21.15.178:0/946895688 conn(0x7ff2f80f1c70 msgr2=0x7ff2f80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:55.949 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.943+0000 7ff2ffed3700 1 -- 172.21.15.178:0/946895688 shutdown_connections 2023-11-15T05:02:55.950 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:55.943+0000 7ff2ffed3700 1 -- 172.21.15.178:0/946895688 wait complete. 2023-11-15T05:02:55.950 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:56.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:56 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:54.379665+0000 mgr.smithi163.qesyrx (mgr.14162) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:56.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:56 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:55.942583+0000 mon.smithi163 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.178:0/946895688' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:57.334 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:57.334 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:58.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:58 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:56.379916+0000 mgr.smithi163.qesyrx (mgr.14162) 232 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:58.521 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.520+0000 7fb2e0ef5700 1 -- 172.21.15.178:0/3062425458 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2dc0f62a0 msgr2=0x7fb2dc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:58.521 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.520+0000 7fb2e0ef5700 1 --2- 172.21.15.178:0/3062425458 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2dc0f62a0 0x7fb2dc0f66c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fb2cc009fb0 tx=0x7fb2cc009b30).stop 2023-11-15T05:02:58.522 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.520+0000 7fb2e0ef5700 1 -- 172.21.15.178:0/3062425458 shutdown_connections 2023-11-15T05:02:58.522 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.520+0000 7fb2e0ef5700 1 --2- 172.21.15.178:0/3062425458 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2dc0f62a0 0x7fb2dc0f66c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:58.522 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.520+0000 7fb2e0ef5700 1 -- 172.21.15.178:0/3062425458 >> 172.21.15.178:0/3062425458 conn(0x7fb2dc0f1aa0 msgr2=0x7fb2dc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:58.522 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.520+0000 7fb2e0ef5700 1 -- 172.21.15.178:0/3062425458 shutdown_connections 2023-11-15T05:02:58.522 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.520+0000 7fb2e0ef5700 1 -- 172.21.15.178:0/3062425458 wait complete. 2023-11-15T05:02:58.523 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.520+0000 7fb2e0ef5700 1 Processor -- start 2023-11-15T05:02:58.525 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.521+0000 7fb2e0ef5700 1 -- start start 2023-11-15T05:02:58.525 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.521+0000 7fb2e0ef5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2dc0f62a0 0x7fb2dc1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:58.525 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.521+0000 7fb2e0ef5700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc1069a0 con 0x7fb2dc0f62a0 2023-11-15T05:02:58.525 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.521+0000 7fb2db7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2dc0f62a0 0x7fb2dc1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:58.525 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.521+0000 7fb2db7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2dc0f62a0 0x7fb2dc1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54434/0 (socket says 172.21.15.178:54434) 2023-11-15T05:02:58.526 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.521+0000 7fb2db7fe700 1 -- 172.21.15.178:0/1339055344 learned_addr learned my addr 172.21.15.178:0/1339055344 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:02:58.526 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.521+0000 7fb2db7fe700 1 -- 172.21.15.178:0/1339055344 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2cc005040 con 0x7fb2dc0f62a0 2023-11-15T05:02:58.526 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.521+0000 7fb2db7fe700 1 --2- 172.21.15.178:0/1339055344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2dc0f62a0 0x7fb2dc1064b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fb2cc013510 tx=0x7fb2cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:58.526 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.522+0000 7fb2c3fff700 1 -- 172.21.15.178:0/1339055344 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2cc009320 con 0x7fb2dc0f62a0 2023-11-15T05:02:58.526 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.522+0000 7fb2e0ef5700 1 -- 172.21.15.178:0/1339055344 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2dc106ba0 con 0x7fb2dc0f62a0 2023-11-15T05:02:58.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.522+0000 7fb2c3fff700 1 -- 172.21.15.178:0/1339055344 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2cc013f00 con 0x7fb2dc0f62a0 2023-11-15T05:02:58.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.522+0000 7fb2c3fff700 1 -- 172.21.15.178:0/1339055344 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2cc014dc0 con 0x7fb2dc0f62a0 2023-11-15T05:02:58.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.522+0000 7fb2e0ef5700 1 -- 172.21.15.178:0/1339055344 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2dc107030 con 0x7fb2dc0f62a0 2023-11-15T05:02:58.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.522+0000 7fb2c3fff700 1 -- 172.21.15.178:0/1339055344 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2cc01d3d0 con 0x7fb2dc0f62a0 2023-11-15T05:02:58.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.523+0000 7fb2c3fff700 1 --2- 172.21.15.178:0/1339055344 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb2c402ffb0 0x7fb2c4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:58.527 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.523+0000 7fb2c3fff700 1 -- 172.21.15.178:0/1339055344 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb2cc041120 con 0x7fb2dc0f62a0 2023-11-15T05:02:58.528 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.523+0000 7fb2e0ef5700 1 -- 172.21.15.178:0/1339055344 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2dc045ad0 con 0x7fb2dc0f62a0 2023-11-15T05:02:58.528 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.523+0000 7fb2daffd700 1 --2- 172.21.15.178:0/1339055344 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb2c402ffb0 0x7fb2c4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:58.528 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.523+0000 7fb2daffd700 1 --2- 172.21.15.178:0/1339055344 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb2c402ffb0 0x7fb2c4032470 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fb2d00096a0 tx=0x7fb2d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:58.529 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.528+0000 7fb2c3fff700 1 -- 172.21.15.178:0/1339055344 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb2cc012920 con 0x7fb2dc0f62a0 2023-11-15T05:02:58.734 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.733+0000 7fb2e0ef5700 1 -- 172.21.15.178:0/1339055344 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb2dc0fbdd0 con 0x7fb2dc0f62a0 2023-11-15T05:02:58.735 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.733+0000 7fb2c3fff700 1 -- 172.21.15.178:0/1339055344 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb2cc02b020 con 0x7fb2dc0f62a0 2023-11-15T05:02:58.735 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:02:58.736 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:58.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.735+0000 7fb2e0ef5700 1 -- 172.21.15.178:0/1339055344 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb2c402ffb0 msgr2=0x7fb2c4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:58.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.735+0000 7fb2e0ef5700 1 --2- 172.21.15.178:0/1339055344 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb2c402ffb0 0x7fb2c4032470 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fb2d00096a0 tx=0x7fb2d0006b40).stop 2023-11-15T05:02:58.737 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.735+0000 7fb2e0ef5700 1 -- 172.21.15.178:0/1339055344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2dc0f62a0 msgr2=0x7fb2dc1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:58.738 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.735+0000 7fb2e0ef5700 1 --2- 172.21.15.178:0/1339055344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2dc0f62a0 0x7fb2dc1064b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fb2cc013510 tx=0x7fb2cc00b040).stop 2023-11-15T05:02:58.738 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.735+0000 7fb2e0ef5700 1 -- 172.21.15.178:0/1339055344 shutdown_connections 2023-11-15T05:02:58.738 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.735+0000 7fb2e0ef5700 1 --2- 172.21.15.178:0/1339055344 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fb2c402ffb0 0x7fb2c4032470 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:58.739 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.735+0000 7fb2e0ef5700 1 --2- 172.21.15.178:0/1339055344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2dc0f62a0 0x7fb2dc1064b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:58.739 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.735+0000 7fb2e0ef5700 1 -- 172.21.15.178:0/1339055344 >> 172.21.15.178:0/1339055344 conn(0x7fb2dc0f1aa0 msgr2=0x7fb2dc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:58.739 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.735+0000 7fb2e0ef5700 1 -- 172.21.15.178:0/1339055344 shutdown_connections 2023-11-15T05:02:58.739 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:02:58.735+0000 7fb2e0ef5700 1 -- 172.21.15.178:0/1339055344 wait complete. 2023-11-15T05:02:58.740 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:02:59.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:02:59 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:02:58.735087+0000 mon.smithi163 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.178:0/1339055344' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:00.127 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:00.128 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:00.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:00 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:02:58.380178+0000 mgr.smithi163.qesyrx (mgr.14162) 233 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:01.326 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.324+0000 7f3a10d44700 1 -- 172.21.15.178:0/394132013 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a0c0f51a0 msgr2=0x7f3a0c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:01.326 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.324+0000 7f3a10d44700 1 --2- 172.21.15.178:0/394132013 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a0c0f51a0 0x7f3a0c0f55c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f3a00009fb0 tx=0x7f3a00009b30).stop 2023-11-15T05:03:01.326 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.324+0000 7f3a10d44700 1 -- 172.21.15.178:0/394132013 shutdown_connections 2023-11-15T05:03:01.327 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.324+0000 7f3a10d44700 1 --2- 172.21.15.178:0/394132013 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a0c0f51a0 0x7f3a0c0f55c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:01.327 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.324+0000 7f3a10d44700 1 -- 172.21.15.178:0/394132013 >> 172.21.15.178:0/394132013 conn(0x7f3a0c0f09a0 msgr2=0x7f3a0c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:01.327 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.324+0000 7f3a10d44700 1 -- 172.21.15.178:0/394132013 shutdown_connections 2023-11-15T05:03:01.327 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.324+0000 7f3a10d44700 1 -- 172.21.15.178:0/394132013 wait complete. 2023-11-15T05:03:01.327 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.324+0000 7f3a10d44700 1 Processor -- start 2023-11-15T05:03:01.328 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.324+0000 7f3a10d44700 1 -- start start 2023-11-15T05:03:01.328 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.325+0000 7f3a10d44700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a0c0f51a0 0x7f3a0c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:01.328 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.325+0000 7f3a10d44700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c060240 con 0x7f3a0c0f51a0 2023-11-15T05:03:01.329 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.325+0000 7f3a0b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a0c0f51a0 0x7f3a0c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:01.330 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.325+0000 7f3a0b7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a0c0f51a0 0x7f3a0c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54458/0 (socket says 172.21.15.178:54458) 2023-11-15T05:03:01.330 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.325+0000 7f3a0b7fe700 1 -- 172.21.15.178:0/3036616598 learned_addr learned my addr 172.21.15.178:0/3036616598 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:03:01.330 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.325+0000 7f3a0b7fe700 1 -- 172.21.15.178:0/3036616598 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a00005040 con 0x7f3a0c0f51a0 2023-11-15T05:03:01.330 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.325+0000 7f3a0b7fe700 1 --2- 172.21.15.178:0/3036616598 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a0c0f51a0 0x7f3a0c1091b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f3a00013510 tx=0x7f3a0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:01.331 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.325+0000 7f39f3fff700 1 -- 172.21.15.178:0/3036616598 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a00009320 con 0x7f3a0c0f51a0 2023-11-15T05:03:01.331 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.325+0000 7f3a10d44700 1 -- 172.21.15.178:0/3036616598 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a0c060440 con 0x7f3a0c0f51a0 2023-11-15T05:03:01.331 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.325+0000 7f39f3fff700 1 -- 172.21.15.178:0/3036616598 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a00013f00 con 0x7f3a0c0f51a0 2023-11-15T05:03:01.331 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.325+0000 7f39f3fff700 1 -- 172.21.15.178:0/3036616598 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a00014dc0 con 0x7f3a0c0f51a0 2023-11-15T05:03:01.331 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.325+0000 7f3a10d44700 1 -- 172.21.15.178:0/3036616598 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a0c109860 con 0x7f3a0c0f51a0 2023-11-15T05:03:01.332 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.326+0000 7f39f3fff700 1 -- 172.21.15.178:0/3036616598 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3a0001d3d0 con 0x7f3a0c0f51a0 2023-11-15T05:03:01.332 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.326+0000 7f39f3fff700 1 --2- 172.21.15.178:0/3036616598 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f39f402ff60 0x7f39f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:01.332 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.326+0000 7f3a10d44700 1 -- 172.21.15.178:0/3036616598 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a0c045ad0 con 0x7f3a0c0f51a0 2023-11-15T05:03:01.332 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.326+0000 7f39f3fff700 1 -- 172.21.15.178:0/3036616598 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3a00041110 con 0x7f3a0c0f51a0 2023-11-15T05:03:01.333 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.327+0000 7f3a0affd700 1 --2- 172.21.15.178:0/3036616598 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f39f402ff60 0x7f39f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:01.333 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.327+0000 7f3a0affd700 1 --2- 172.21.15.178:0/3036616598 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f39f402ff60 0x7f39f4032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f39fc0096a0 tx=0x7f39fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:01.333 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.331+0000 7f39f3fff700 1 -- 172.21.15.178:0/3036616598 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a00012900 con 0x7f3a0c0f51a0 2023-11-15T05:03:01.536 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.535+0000 7f3a10d44700 1 -- 172.21.15.178:0/3036616598 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a0c10a080 con 0x7f3a0c0f51a0 2023-11-15T05:03:01.536 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.535+0000 7f39f3fff700 1 -- 172.21.15.178:0/3036616598 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3a0002b020 con 0x7f3a0c0f51a0 2023-11-15T05:03:01.537 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:03:01.537 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:01.538 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.537+0000 7f3a10d44700 1 -- 172.21.15.178:0/3036616598 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f39f402ff60 msgr2=0x7f39f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:01.538 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.537+0000 7f3a10d44700 1 --2- 172.21.15.178:0/3036616598 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f39f402ff60 0x7f39f4032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f39fc0096a0 tx=0x7f39fc006b40).stop 2023-11-15T05:03:01.539 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.537+0000 7f3a10d44700 1 -- 172.21.15.178:0/3036616598 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a0c0f51a0 msgr2=0x7f3a0c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:01.539 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.537+0000 7f3a10d44700 1 --2- 172.21.15.178:0/3036616598 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a0c0f51a0 0x7f3a0c1091b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f3a00013510 tx=0x7f3a0000b040).stop 2023-11-15T05:03:01.539 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.537+0000 7f3a10d44700 1 -- 172.21.15.178:0/3036616598 shutdown_connections 2023-11-15T05:03:01.539 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.537+0000 7f3a10d44700 1 --2- 172.21.15.178:0/3036616598 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f39f402ff60 0x7f39f4032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:01.539 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.537+0000 7f3a10d44700 1 --2- 172.21.15.178:0/3036616598 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a0c0f51a0 0x7f3a0c1091b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:01.540 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.537+0000 7f3a10d44700 1 -- 172.21.15.178:0/3036616598 >> 172.21.15.178:0/3036616598 conn(0x7f3a0c0f09a0 msgr2=0x7f3a0c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:01.540 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.537+0000 7f3a10d44700 1 -- 172.21.15.178:0/3036616598 shutdown_connections 2023-11-15T05:03:01.540 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:01.537+0000 7f3a10d44700 1 -- 172.21.15.178:0/3036616598 wait complete. 2023-11-15T05:03:01.540 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:03:02.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:02 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:00.380499+0000 mgr.smithi163.qesyrx (mgr.14162) 234 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:02.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:02 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:01.537045+0000 mon.smithi163 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.178:0/3036616598' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:02.917 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:02.918 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:04.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.045+0000 7fdc59c90700 1 -- 172.21.15.178:0/2400481304 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc54060280 msgr2=0x7fdc540f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:04.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.045+0000 7fdc59c90700 1 --2- 172.21.15.178:0/2400481304 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc54060280 0x7fdc540f90f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fdc44009fb0 tx=0x7fdc44009b30).stop 2023-11-15T05:03:04.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.045+0000 7fdc59c90700 1 -- 172.21.15.178:0/2400481304 shutdown_connections 2023-11-15T05:03:04.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.045+0000 7fdc59c90700 1 --2- 172.21.15.178:0/2400481304 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc54060280 0x7fdc540f90f0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:04.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.045+0000 7fdc59c90700 1 -- 172.21.15.178:0/2400481304 >> 172.21.15.178:0/2400481304 conn(0x7fdc540f0b70 msgr2=0x7fdc540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:04.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.046+0000 7fdc59c90700 1 -- 172.21.15.178:0/2400481304 shutdown_connections 2023-11-15T05:03:04.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.046+0000 7fdc59c90700 1 -- 172.21.15.178:0/2400481304 wait complete. 2023-11-15T05:03:04.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.046+0000 7fdc59c90700 1 Processor -- start 2023-11-15T05:03:04.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.046+0000 7fdc59c90700 1 -- start start 2023-11-15T05:03:04.050 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.047+0000 7fdc59c90700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc54060280 0x7fdc541066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:04.050 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.047+0000 7fdc59c90700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc54106bd0 con 0x7fdc54060280 2023-11-15T05:03:04.051 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.047+0000 7fdc537fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc54060280 0x7fdc541066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:04.052 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.047+0000 7fdc537fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc54060280 0x7fdc541066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54482/0 (socket says 172.21.15.178:54482) 2023-11-15T05:03:04.052 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.047+0000 7fdc537fe700 1 -- 172.21.15.178:0/2562677810 learned_addr learned my addr 172.21.15.178:0/2562677810 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:03:04.052 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.047+0000 7fdc537fe700 1 -- 172.21.15.178:0/2562677810 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc44005040 con 0x7fdc54060280 2023-11-15T05:03:04.053 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.047+0000 7fdc537fe700 1 --2- 172.21.15.178:0/2562677810 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc54060280 0x7fdc541066e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fdc44013510 tx=0x7fdc4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:04.053 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.048+0000 7fdc3bfff700 1 -- 172.21.15.178:0/2562677810 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc44009320 con 0x7fdc54060280 2023-11-15T05:03:04.053 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.048+0000 7fdc59c90700 1 -- 172.21.15.178:0/2562677810 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc54106dd0 con 0x7fdc54060280 2023-11-15T05:03:04.053 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.048+0000 7fdc3bfff700 1 -- 172.21.15.178:0/2562677810 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc44013f00 con 0x7fdc54060280 2023-11-15T05:03:04.053 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.048+0000 7fdc3bfff700 1 -- 172.21.15.178:0/2562677810 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc44014dc0 con 0x7fdc54060280 2023-11-15T05:03:04.054 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.048+0000 7fdc59c90700 1 -- 172.21.15.178:0/2562677810 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc54107260 con 0x7fdc54060280 2023-11-15T05:03:04.054 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.048+0000 7fdc3bfff700 1 -- 172.21.15.178:0/2562677810 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc440095f0 con 0x7fdc54060280 2023-11-15T05:03:04.054 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.049+0000 7fdc59c90700 1 -- 172.21.15.178:0/2562677810 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc54045ad0 con 0x7fdc54060280 2023-11-15T05:03:04.054 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.049+0000 7fdc3bfff700 1 --2- 172.21.15.178:0/2562677810 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdc3c02ff60 0x7fdc3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:04.054 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.049+0000 7fdc3bfff700 1 -- 172.21.15.178:0/2562677810 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc44041000 con 0x7fdc54060280 2023-11-15T05:03:04.055 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.049+0000 7fdc52ffd700 1 --2- 172.21.15.178:0/2562677810 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdc3c02ff60 0x7fdc3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:04.055 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.049+0000 7fdc52ffd700 1 --2- 172.21.15.178:0/2562677810 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdc3c02ff60 0x7fdc3c032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fdc480096a0 tx=0x7fdc48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:04.055 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.053+0000 7fdc3bfff700 1 -- 172.21.15.178:0/2562677810 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc4406d2b0 con 0x7fdc54060280 2023-11-15T05:03:04.261 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.259+0000 7fdc59c90700 1 -- 172.21.15.178:0/2562677810 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc54108110 con 0x7fdc54060280 2023-11-15T05:03:04.261 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.260+0000 7fdc3bfff700 1 -- 172.21.15.178:0/2562677810 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdc44010230 con 0x7fdc54060280 2023-11-15T05:03:04.262 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:03:04.262 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:04.263 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.261+0000 7fdc59c90700 1 -- 172.21.15.178:0/2562677810 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdc3c02ff60 msgr2=0x7fdc3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:04.263 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.261+0000 7fdc59c90700 1 --2- 172.21.15.178:0/2562677810 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdc3c02ff60 0x7fdc3c032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fdc480096a0 tx=0x7fdc48006b40).stop 2023-11-15T05:03:04.263 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.261+0000 7fdc59c90700 1 -- 172.21.15.178:0/2562677810 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc54060280 msgr2=0x7fdc541066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:04.263 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.261+0000 7fdc59c90700 1 --2- 172.21.15.178:0/2562677810 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc54060280 0x7fdc541066e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fdc44013510 tx=0x7fdc4400b040).stop 2023-11-15T05:03:04.263 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.261+0000 7fdc59c90700 1 -- 172.21.15.178:0/2562677810 shutdown_connections 2023-11-15T05:03:04.264 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.261+0000 7fdc59c90700 1 --2- 172.21.15.178:0/2562677810 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdc3c02ff60 0x7fdc3c032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:04.264 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.261+0000 7fdc59c90700 1 --2- 172.21.15.178:0/2562677810 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc54060280 0x7fdc541066e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:04.264 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.261+0000 7fdc59c90700 1 -- 172.21.15.178:0/2562677810 >> 172.21.15.178:0/2562677810 conn(0x7fdc540f0b70 msgr2=0x7fdc540f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:04.264 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.261+0000 7fdc59c90700 1 -- 172.21.15.178:0/2562677810 shutdown_connections 2023-11-15T05:03:04.264 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:04.261+0000 7fdc59c90700 1 -- 172.21.15.178:0/2562677810 wait complete. 2023-11-15T05:03:04.265 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:03:04.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:04 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:02.380822+0000 mgr.smithi163.qesyrx (mgr.14162) 235 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:05.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:05 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:04.261674+0000 mon.smithi163 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.178:0/2562677810' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:05.704 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:05.705 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:06.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:06 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:04.381098+0000 mgr.smithi163.qesyrx (mgr.14162) 236 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:06.877 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.872+0000 7f5c88bcd700 1 -- 172.21.15.178:0/2098410233 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c840f6510 msgr2=0x7f5c840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:06.878 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.872+0000 7f5c88bcd700 1 --2- 172.21.15.178:0/2098410233 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c840f6510 0x7f5c840f6930 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f5c74009fb0 tx=0x7f5c74009b30).stop 2023-11-15T05:03:06.878 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.873+0000 7f5c88bcd700 1 -- 172.21.15.178:0/2098410233 shutdown_connections 2023-11-15T05:03:06.879 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.873+0000 7f5c88bcd700 1 --2- 172.21.15.178:0/2098410233 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c840f6510 0x7f5c840f6930 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:06.879 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.873+0000 7f5c88bcd700 1 -- 172.21.15.178:0/2098410233 >> 172.21.15.178:0/2098410233 conn(0x7f5c840f1c70 msgr2=0x7f5c840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:06.879 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.873+0000 7f5c88bcd700 1 -- 172.21.15.178:0/2098410233 shutdown_connections 2023-11-15T05:03:06.880 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.873+0000 7f5c88bcd700 1 -- 172.21.15.178:0/2098410233 wait complete. 2023-11-15T05:03:06.880 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.873+0000 7f5c88bcd700 1 Processor -- start 2023-11-15T05:03:06.880 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.873+0000 7f5c88bcd700 1 -- start start 2023-11-15T05:03:06.881 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.873+0000 7f5c88bcd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c840f6510 0x7f5c84106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:06.881 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.873+0000 7f5c88bcd700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c84106c20 con 0x7f5c840f6510 2023-11-15T05:03:06.882 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.874+0000 7f5c8259c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c840f6510 0x7f5c84106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:06.882 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.874+0000 7f5c8259c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c840f6510 0x7f5c84106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54734/0 (socket says 172.21.15.178:54734) 2023-11-15T05:03:06.883 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.874+0000 7f5c8259c700 1 -- 172.21.15.178:0/1486292980 learned_addr learned my addr 172.21.15.178:0/1486292980 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:03:06.883 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.874+0000 7f5c8259c700 1 -- 172.21.15.178:0/1486292980 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c74005040 con 0x7f5c840f6510 2023-11-15T05:03:06.884 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.874+0000 7f5c8259c700 1 --2- 172.21.15.178:0/1486292980 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c840f6510 0x7f5c84106730 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f5c74013510 tx=0x7f5c7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:06.884 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.874+0000 7f5c7affd700 1 -- 172.21.15.178:0/1486292980 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c74009320 con 0x7f5c840f6510 2023-11-15T05:03:06.885 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.874+0000 7f5c88bcd700 1 -- 172.21.15.178:0/1486292980 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c84106e20 con 0x7f5c840f6510 2023-11-15T05:03:06.885 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.874+0000 7f5c7affd700 1 -- 172.21.15.178:0/1486292980 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c74013f00 con 0x7f5c840f6510 2023-11-15T05:03:06.885 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.874+0000 7f5c7affd700 1 -- 172.21.15.178:0/1486292980 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c74014d90 con 0x7f5c840f6510 2023-11-15T05:03:06.888 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.874+0000 7f5c88bcd700 1 -- 172.21.15.178:0/1486292980 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c841072b0 con 0x7f5c840f6510 2023-11-15T05:03:06.888 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.875+0000 7f5c7affd700 1 -- 172.21.15.178:0/1486292980 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5c74014420 con 0x7f5c840f6510 2023-11-15T05:03:06.889 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.875+0000 7f5c88bcd700 1 -- 172.21.15.178:0/1486292980 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c84045ad0 con 0x7f5c840f6510 2023-11-15T05:03:06.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.875+0000 7f5c7affd700 1 --2- 172.21.15.178:0/1486292980 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5c7002ff60 0x7f5c70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:06.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.875+0000 7f5c7affd700 1 -- 172.21.15.178:0/1486292980 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5c74041110 con 0x7f5c840f6510 2023-11-15T05:03:06.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.875+0000 7f5c81d9b700 1 --2- 172.21.15.178:0/1486292980 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5c7002ff60 0x7f5c70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:06.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.876+0000 7f5c81d9b700 1 --2- 172.21.15.178:0/1486292980 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5c7002ff60 0x7f5c70032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f5c6c0096a0 tx=0x7f5c6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:06.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:06.880+0000 7f5c7affd700 1 -- 172.21.15.178:0/1486292980 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c740095f0 con 0x7f5c840f6510 2023-11-15T05:03:07.088 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:07.086+0000 7f5c88bcd700 1 -- 172.21.15.178:0/1486292980 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c840fbf60 con 0x7f5c840f6510 2023-11-15T05:03:07.088 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:07.087+0000 7f5c7affd700 1 -- 172.21.15.178:0/1486292980 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5c7401db70 con 0x7f5c840f6510 2023-11-15T05:03:07.089 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:03:07.089 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:07.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:07.088+0000 7f5c88bcd700 1 -- 172.21.15.178:0/1486292980 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5c7002ff60 msgr2=0x7f5c70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:07.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:07.088+0000 7f5c88bcd700 1 --2- 172.21.15.178:0/1486292980 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5c7002ff60 0x7f5c70032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f5c6c0096a0 tx=0x7f5c6c006b40).stop 2023-11-15T05:03:07.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:07.088+0000 7f5c88bcd700 1 -- 172.21.15.178:0/1486292980 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c840f6510 msgr2=0x7f5c84106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:07.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:07.088+0000 7f5c88bcd700 1 --2- 172.21.15.178:0/1486292980 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c840f6510 0x7f5c84106730 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f5c74013510 tx=0x7f5c7400b040).stop 2023-11-15T05:03:07.091 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:07.088+0000 7f5c88bcd700 1 -- 172.21.15.178:0/1486292980 shutdown_connections 2023-11-15T05:03:07.091 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:07.088+0000 7f5c88bcd700 1 --2- 172.21.15.178:0/1486292980 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f5c7002ff60 0x7f5c70032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:07.091 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:07.088+0000 7f5c88bcd700 1 --2- 172.21.15.178:0/1486292980 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c840f6510 0x7f5c84106730 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:07.091 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:07.088+0000 7f5c88bcd700 1 -- 172.21.15.178:0/1486292980 >> 172.21.15.178:0/1486292980 conn(0x7f5c840f1c70 msgr2=0x7f5c840f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:07.091 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:07.088+0000 7f5c88bcd700 1 -- 172.21.15.178:0/1486292980 shutdown_connections 2023-11-15T05:03:07.092 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:07.088+0000 7f5c88bcd700 1 -- 172.21.15.178:0/1486292980 wait complete. 2023-11-15T05:03:07.092 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:03:08.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:08 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:06.381471+0000 mgr.smithi163.qesyrx (mgr.14162) 237 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:08.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:08 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:07.088412+0000 mon.smithi163 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.178:0/1486292980' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:08.470 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:08.471 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:09.663 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.660+0000 7ff0ff116700 1 -- 172.21.15.178:0/2229389966 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0f80f6510 msgr2=0x7ff0f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:09.664 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.660+0000 7ff0ff116700 1 --2- 172.21.15.178:0/2229389966 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0f80f6510 0x7ff0f80f6930 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7ff0ec009fb0 tx=0x7ff0ec009b30).stop 2023-11-15T05:03:09.664 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.660+0000 7ff0ff116700 1 -- 172.21.15.178:0/2229389966 shutdown_connections 2023-11-15T05:03:09.664 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.660+0000 7ff0ff116700 1 --2- 172.21.15.178:0/2229389966 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0f80f6510 0x7ff0f80f6930 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:09.664 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.660+0000 7ff0ff116700 1 -- 172.21.15.178:0/2229389966 >> 172.21.15.178:0/2229389966 conn(0x7ff0f80f1c70 msgr2=0x7ff0f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:09.665 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.660+0000 7ff0ff116700 1 -- 172.21.15.178:0/2229389966 shutdown_connections 2023-11-15T05:03:09.665 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.660+0000 7ff0ff116700 1 -- 172.21.15.178:0/2229389966 wait complete. 2023-11-15T05:03:09.665 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.661+0000 7ff0ff116700 1 Processor -- start 2023-11-15T05:03:09.665 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.661+0000 7ff0ff116700 1 -- start start 2023-11-15T05:03:09.665 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.661+0000 7ff0ff116700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0f80f6510 0x7ff0f8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:09.666 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.661+0000 7ff0ff116700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f8106c20 con 0x7ff0f80f6510 2023-11-15T05:03:09.666 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.661+0000 7ff0fceb2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0f80f6510 0x7ff0f8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:09.666 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.661+0000 7ff0fceb2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0f80f6510 0x7ff0f8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54744/0 (socket says 172.21.15.178:54744) 2023-11-15T05:03:09.666 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.661+0000 7ff0fceb2700 1 -- 172.21.15.178:0/480444365 learned_addr learned my addr 172.21.15.178:0/480444365 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:03:09.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.662+0000 7ff0fceb2700 1 -- 172.21.15.178:0/480444365 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ec005040 con 0x7ff0f80f6510 2023-11-15T05:03:09.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.662+0000 7ff0fceb2700 1 --2- 172.21.15.178:0/480444365 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0f80f6510 0x7ff0f8106730 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7ff0ec013510 tx=0x7ff0ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:09.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.662+0000 7ff0f57fa700 1 -- 172.21.15.178:0/480444365 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0ec009320 con 0x7ff0f80f6510 2023-11-15T05:03:09.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.662+0000 7ff0ff116700 1 -- 172.21.15.178:0/480444365 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f8106e20 con 0x7ff0f80f6510 2023-11-15T05:03:09.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.662+0000 7ff0f57fa700 1 -- 172.21.15.178:0/480444365 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff0ec013f00 con 0x7ff0f80f6510 2023-11-15T05:03:09.668 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.662+0000 7ff0f57fa700 1 -- 172.21.15.178:0/480444365 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0ec014d90 con 0x7ff0f80f6510 2023-11-15T05:03:09.668 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.662+0000 7ff0ff116700 1 -- 172.21.15.178:0/480444365 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f81072b0 con 0x7ff0f80f6510 2023-11-15T05:03:09.669 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.663+0000 7ff0ff116700 1 -- 172.21.15.178:0/480444365 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f8045ad0 con 0x7ff0f80f6510 2023-11-15T05:03:09.669 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.663+0000 7ff0f57fa700 1 -- 172.21.15.178:0/480444365 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff0ec0095f0 con 0x7ff0f80f6510 2023-11-15T05:03:09.669 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.663+0000 7ff0f57fa700 1 --2- 172.21.15.178:0/480444365 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff0e802ff60 0x7ff0e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:09.670 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.663+0000 7ff0f57fa700 1 -- 172.21.15.178:0/480444365 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff0ec041000 con 0x7ff0f80f6510 2023-11-15T05:03:09.670 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.663+0000 7ff0f7fff700 1 --2- 172.21.15.178:0/480444365 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff0e802ff60 0x7ff0e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:09.670 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.664+0000 7ff0f7fff700 1 --2- 172.21.15.178:0/480444365 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff0e802ff60 0x7ff0e8032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7ff0e40096a0 tx=0x7ff0e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:09.670 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.668+0000 7ff0f57fa700 1 -- 172.21.15.178:0/480444365 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff0ec014420 con 0x7ff0f80f6510 2023-11-15T05:03:09.872 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.871+0000 7ff0ff116700 1 -- 172.21.15.178:0/480444365 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff0f80fbf60 con 0x7ff0f80f6510 2023-11-15T05:03:09.873 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.872+0000 7ff0f57fa700 1 -- 172.21.15.178:0/480444365 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff0ec01d7f0 con 0x7ff0f80f6510 2023-11-15T05:03:09.873 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:03:09.873 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:09.874 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.873+0000 7ff0ff116700 1 -- 172.21.15.178:0/480444365 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff0e802ff60 msgr2=0x7ff0e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:09.875 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.873+0000 7ff0ff116700 1 --2- 172.21.15.178:0/480444365 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff0e802ff60 0x7ff0e8032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7ff0e40096a0 tx=0x7ff0e4006b40).stop 2023-11-15T05:03:09.875 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.873+0000 7ff0ff116700 1 -- 172.21.15.178:0/480444365 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0f80f6510 msgr2=0x7ff0f8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:09.875 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.873+0000 7ff0ff116700 1 --2- 172.21.15.178:0/480444365 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0f80f6510 0x7ff0f8106730 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7ff0ec013510 tx=0x7ff0ec00b040).stop 2023-11-15T05:03:09.875 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.873+0000 7ff0ff116700 1 -- 172.21.15.178:0/480444365 shutdown_connections 2023-11-15T05:03:09.875 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.873+0000 7ff0ff116700 1 --2- 172.21.15.178:0/480444365 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7ff0e802ff60 0x7ff0e8032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:09.876 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.873+0000 7ff0ff116700 1 --2- 172.21.15.178:0/480444365 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0f80f6510 0x7ff0f8106730 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:09.876 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.873+0000 7ff0ff116700 1 -- 172.21.15.178:0/480444365 >> 172.21.15.178:0/480444365 conn(0x7ff0f80f1c70 msgr2=0x7ff0f80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:09.876 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.873+0000 7ff0ff116700 1 -- 172.21.15.178:0/480444365 shutdown_connections 2023-11-15T05:03:09.876 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:09.873+0000 7ff0ff116700 1 -- 172.21.15.178:0/480444365 wait complete. 2023-11-15T05:03:09.877 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:03:10.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:10 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:08.381814+0000 mgr.smithi163.qesyrx (mgr.14162) 238 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:10.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:10 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:09.873236+0000 mon.smithi163 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.178:0/480444365' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:11.265 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:11.266 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:12.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:12 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:10.382224+0000 mgr.smithi163.qesyrx (mgr.14162) 239 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:12.459 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.454+0000 7fcca90c8700 1 -- 172.21.15.178:0/1118829072 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcca40f5410 msgr2=0x7fcca40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:12.459 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.454+0000 7fcca90c8700 1 --2- 172.21.15.178:0/1118829072 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcca40f5410 0x7fcca40f5830 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fcc8c009fb0 tx=0x7fcc8c009b30).stop 2023-11-15T05:03:12.460 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.454+0000 7fcca90c8700 1 -- 172.21.15.178:0/1118829072 shutdown_connections 2023-11-15T05:03:12.460 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.454+0000 7fcca90c8700 1 --2- 172.21.15.178:0/1118829072 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcca40f5410 0x7fcca40f5830 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:12.460 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.454+0000 7fcca90c8700 1 -- 172.21.15.178:0/1118829072 >> 172.21.15.178:0/1118829072 conn(0x7fcca40f0b70 msgr2=0x7fcca40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:12.461 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.454+0000 7fcca90c8700 1 -- 172.21.15.178:0/1118829072 shutdown_connections 2023-11-15T05:03:12.461 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.454+0000 7fcca90c8700 1 -- 172.21.15.178:0/1118829072 wait complete. 2023-11-15T05:03:12.461 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.455+0000 7fcca90c8700 1 Processor -- start 2023-11-15T05:03:12.462 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.455+0000 7fcca90c8700 1 -- start start 2023-11-15T05:03:12.462 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.455+0000 7fcca90c8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcca40f5410 0x7fcca41055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:12.462 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.455+0000 7fcca90c8700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca4105ac0 con 0x7fcca40f5410 2023-11-15T05:03:12.463 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.455+0000 7fcca2d9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcca40f5410 0x7fcca41055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:12.463 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.455+0000 7fcca2d9d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcca40f5410 0x7fcca41055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54770/0 (socket says 172.21.15.178:54770) 2023-11-15T05:03:12.464 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.455+0000 7fcca2d9d700 1 -- 172.21.15.178:0/3498856731 learned_addr learned my addr 172.21.15.178:0/3498856731 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:03:12.464 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.456+0000 7fcca2d9d700 1 -- 172.21.15.178:0/3498856731 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8c005040 con 0x7fcca40f5410 2023-11-15T05:03:12.464 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.456+0000 7fcca2d9d700 1 --2- 172.21.15.178:0/3498856731 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcca40f5410 0x7fcca41055d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fcc8c014000 tx=0x7fcc8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:12.464 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.456+0000 7fcc9b7fe700 1 -- 172.21.15.178:0/3498856731 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc8c0095f0 con 0x7fcca40f5410 2023-11-15T05:03:12.464 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.456+0000 7fcca90c8700 1 -- 172.21.15.178:0/3498856731 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca4105cc0 con 0x7fcca40f5410 2023-11-15T05:03:12.464 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.456+0000 7fcc9b7fe700 1 -- 172.21.15.178:0/3498856731 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc8c007e70 con 0x7fcca40f5410 2023-11-15T05:03:12.465 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.456+0000 7fcc9b7fe700 1 -- 172.21.15.178:0/3498856731 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc8c014ea0 con 0x7fcca40f5410 2023-11-15T05:03:12.465 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.456+0000 7fcca90c8700 1 -- 172.21.15.178:0/3498856731 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca4106150 con 0x7fcca40f5410 2023-11-15T05:03:12.465 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.457+0000 7fcc9b7fe700 1 -- 172.21.15.178:0/3498856731 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcc8c014690 con 0x7fcca40f5410 2023-11-15T05:03:12.465 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.457+0000 7fcca90c8700 1 -- 172.21.15.178:0/3498856731 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca4045ad0 con 0x7fcca40f5410 2023-11-15T05:03:12.465 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.457+0000 7fcc9b7fe700 1 --2- 172.21.15.178:0/3498856731 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcc9002ffb0 0x7fcc90032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:12.466 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.457+0000 7fcc9b7fe700 1 -- 172.21.15.178:0/3498856731 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcc8c040830 con 0x7fcca40f5410 2023-11-15T05:03:12.466 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.457+0000 7fcca259c700 1 --2- 172.21.15.178:0/3498856731 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcc9002ffb0 0x7fcc90032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:12.466 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.458+0000 7fcca259c700 1 --2- 172.21.15.178:0/3498856731 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcc9002ffb0 0x7fcc90032470 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fcc940096a0 tx=0x7fcc94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:12.467 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.462+0000 7fcc9b7fe700 1 -- 172.21.15.178:0/3498856731 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc8c00f300 con 0x7fcca40f5410 2023-11-15T05:03:12.669 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.666+0000 7fcca90c8700 1 -- 172.21.15.178:0/3498856731 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcca4107000 con 0x7fcca40f5410 2023-11-15T05:03:12.669 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.667+0000 7fcc9b7fe700 1 -- 172.21.15.178:0/3498856731 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcc8c00f300 con 0x7fcca40f5410 2023-11-15T05:03:12.670 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:03:12.670 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:12.671 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.669+0000 7fcca90c8700 1 -- 172.21.15.178:0/3498856731 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcc9002ffb0 msgr2=0x7fcc90032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:12.671 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.669+0000 7fcca90c8700 1 --2- 172.21.15.178:0/3498856731 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcc9002ffb0 0x7fcc90032470 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fcc940096a0 tx=0x7fcc94006b40).stop 2023-11-15T05:03:12.672 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.669+0000 7fcca90c8700 1 -- 172.21.15.178:0/3498856731 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcca40f5410 msgr2=0x7fcca41055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:12.672 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.669+0000 7fcca90c8700 1 --2- 172.21.15.178:0/3498856731 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcca40f5410 0x7fcca41055d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fcc8c014000 tx=0x7fcc8c00b040).stop 2023-11-15T05:03:12.672 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.669+0000 7fcca90c8700 1 -- 172.21.15.178:0/3498856731 shutdown_connections 2023-11-15T05:03:12.672 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.669+0000 7fcca90c8700 1 --2- 172.21.15.178:0/3498856731 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcc9002ffb0 0x7fcc90032470 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:12.672 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.669+0000 7fcca90c8700 1 --2- 172.21.15.178:0/3498856731 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcca40f5410 0x7fcca41055d0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:12.673 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.669+0000 7fcca90c8700 1 -- 172.21.15.178:0/3498856731 >> 172.21.15.178:0/3498856731 conn(0x7fcca40f0b70 msgr2=0x7fcca40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:12.673 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.669+0000 7fcca90c8700 1 -- 172.21.15.178:0/3498856731 shutdown_connections 2023-11-15T05:03:12.673 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:12.669+0000 7fcca90c8700 1 -- 172.21.15.178:0/3498856731 wait complete. 2023-11-15T05:03:12.673 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:03:13.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:13 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:12.668905+0000 mon.smithi163 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.178:0/3498856731' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:14.099 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:14.100 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:14.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:14 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:12.382602+0000 mgr.smithi163.qesyrx (mgr.14162) 240 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:15.333 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.332+0000 7f4e713cc700 1 -- 172.21.15.178:0/351824200 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e6c0f5410 msgr2=0x7f4e6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:15.334 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.332+0000 7f4e713cc700 1 --2- 172.21.15.178:0/351824200 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e6c0f5410 0x7f4e6c0f5830 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f4e60009fb0 tx=0x7f4e60009b30).stop 2023-11-15T05:03:15.334 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.332+0000 7f4e713cc700 1 -- 172.21.15.178:0/351824200 shutdown_connections 2023-11-15T05:03:15.334 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.332+0000 7f4e713cc700 1 --2- 172.21.15.178:0/351824200 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e6c0f5410 0x7f4e6c0f5830 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:15.334 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.332+0000 7f4e713cc700 1 -- 172.21.15.178:0/351824200 >> 172.21.15.178:0/351824200 conn(0x7f4e6c0f0b70 msgr2=0x7f4e6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:15.335 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.332+0000 7f4e713cc700 1 -- 172.21.15.178:0/351824200 shutdown_connections 2023-11-15T05:03:15.335 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.332+0000 7f4e713cc700 1 -- 172.21.15.178:0/351824200 wait complete. 2023-11-15T05:03:15.337 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.332+0000 7f4e713cc700 1 Processor -- start 2023-11-15T05:03:15.337 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.333+0000 7f4e713cc700 1 -- start start 2023-11-15T05:03:15.338 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.333+0000 7f4e713cc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e6c0f5410 0x7f4e6c106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:15.338 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.333+0000 7f4e713cc700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c106c40 con 0x7f4e6c0f5410 2023-11-15T05:03:15.338 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.333+0000 7f4e6affd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e6c0f5410 0x7f4e6c106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:15.338 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.333+0000 7f4e6affd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e6c0f5410 0x7f4e6c106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:41218/0 (socket says 172.21.15.178:41218) 2023-11-15T05:03:15.339 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.333+0000 7f4e6affd700 1 -- 172.21.15.178:0/422113487 learned_addr learned my addr 172.21.15.178:0/422113487 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:03:15.339 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.333+0000 7f4e6affd700 1 -- 172.21.15.178:0/422113487 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e60005040 con 0x7f4e6c0f5410 2023-11-15T05:03:15.339 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.334+0000 7f4e6affd700 1 --2- 172.21.15.178:0/422113487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e6c0f5410 0x7f4e6c106750 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f4e60009f80 tx=0x7f4e6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:15.339 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.334+0000 7f4e537fe700 1 -- 172.21.15.178:0/422113487 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e60009320 con 0x7f4e6c0f5410 2023-11-15T05:03:15.339 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.334+0000 7f4e713cc700 1 -- 172.21.15.178:0/422113487 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e6c106e40 con 0x7f4e6c0f5410 2023-11-15T05:03:15.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.334+0000 7f4e537fe700 1 -- 172.21.15.178:0/422113487 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e60013ec0 con 0x7f4e6c0f5410 2023-11-15T05:03:15.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.334+0000 7f4e537fe700 1 -- 172.21.15.178:0/422113487 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e6001ece0 con 0x7f4e6c0f5410 2023-11-15T05:03:15.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.334+0000 7f4e713cc700 1 -- 172.21.15.178:0/422113487 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e6c1072d0 con 0x7f4e6c0f5410 2023-11-15T05:03:15.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.335+0000 7f4e537fe700 1 -- 172.21.15.178:0/422113487 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4e6001e420 con 0x7f4e6c0f5410 2023-11-15T05:03:15.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.335+0000 7f4e713cc700 1 -- 172.21.15.178:0/422113487 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e6c045ad0 con 0x7f4e6c0f5410 2023-11-15T05:03:15.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.335+0000 7f4e537fe700 1 --2- 172.21.15.178:0/422113487 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4e5402ff60 0x7f4e54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:15.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.335+0000 7f4e537fe700 1 -- 172.21.15.178:0/422113487 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4e60040e50 con 0x7f4e6c0f5410 2023-11-15T05:03:15.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.335+0000 7f4e6a7fc700 1 --2- 172.21.15.178:0/422113487 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4e5402ff60 0x7f4e54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:15.341 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.335+0000 7f4e6a7fc700 1 --2- 172.21.15.178:0/422113487 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4e5402ff60 0x7f4e54032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f4e5c0096a0 tx=0x7f4e5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:15.342 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.340+0000 7f4e537fe700 1 -- 172.21.15.178:0/422113487 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e6001e730 con 0x7f4e6c0f5410 2023-11-15T05:03:15.546 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.544+0000 7f4e713cc700 1 -- 172.21.15.178:0/422113487 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e6c108120 con 0x7f4e6c0f5410 2023-11-15T05:03:15.547 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.545+0000 7f4e537fe700 1 -- 172.21.15.178:0/422113487 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4e6001cb10 con 0x7f4e6c0f5410 2023-11-15T05:03:15.548 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:03:15.548 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:15.550 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.546+0000 7f4e713cc700 1 -- 172.21.15.178:0/422113487 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4e5402ff60 msgr2=0x7f4e54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:15.550 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.546+0000 7f4e713cc700 1 --2- 172.21.15.178:0/422113487 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4e5402ff60 0x7f4e54032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f4e5c0096a0 tx=0x7f4e5c006b40).stop 2023-11-15T05:03:15.551 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.546+0000 7f4e713cc700 1 -- 172.21.15.178:0/422113487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e6c0f5410 msgr2=0x7f4e6c106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:15.551 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.546+0000 7f4e713cc700 1 --2- 172.21.15.178:0/422113487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e6c0f5410 0x7f4e6c106750 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f4e60009f80 tx=0x7f4e6000b040).stop 2023-11-15T05:03:15.552 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.546+0000 7f4e713cc700 1 -- 172.21.15.178:0/422113487 shutdown_connections 2023-11-15T05:03:15.552 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.546+0000 7f4e713cc700 1 --2- 172.21.15.178:0/422113487 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4e5402ff60 0x7f4e54032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:15.552 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.546+0000 7f4e713cc700 1 --2- 172.21.15.178:0/422113487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e6c0f5410 0x7f4e6c106750 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:15.553 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.546+0000 7f4e713cc700 1 -- 172.21.15.178:0/422113487 >> 172.21.15.178:0/422113487 conn(0x7f4e6c0f0b70 msgr2=0x7f4e6c05fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:15.553 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.546+0000 7f4e713cc700 1 -- 172.21.15.178:0/422113487 shutdown_connections 2023-11-15T05:03:15.553 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:15.547+0000 7f4e713cc700 1 -- 172.21.15.178:0/422113487 wait complete. 2023-11-15T05:03:15.554 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:03:16.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:14.382837+0000 mgr.smithi163.qesyrx (mgr.14162) 241 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:16.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:16 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:15.546376+0000 mon.smithi163 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.178:0/422113487' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:16.976 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:16.977 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:18.202 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.200+0000 7f78d2586700 1 -- 172.21.15.178:0/1180892451 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78cc0f5410 msgr2=0x7f78cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:18.203 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.200+0000 7f78d2586700 1 --2- 172.21.15.178:0/1180892451 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78cc0f5410 0x7f78cc0f5830 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f78b4009fb0 tx=0x7f78b4009b30).stop 2023-11-15T05:03:18.203 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.200+0000 7f78d2586700 1 -- 172.21.15.178:0/1180892451 shutdown_connections 2023-11-15T05:03:18.203 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.200+0000 7f78d2586700 1 --2- 172.21.15.178:0/1180892451 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78cc0f5410 0x7f78cc0f5830 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:18.203 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.200+0000 7f78d2586700 1 -- 172.21.15.178:0/1180892451 >> 172.21.15.178:0/1180892451 conn(0x7f78cc0f0b70 msgr2=0x7f78cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:18.203 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.200+0000 7f78d2586700 1 -- 172.21.15.178:0/1180892451 shutdown_connections 2023-11-15T05:03:18.204 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.200+0000 7f78d2586700 1 -- 172.21.15.178:0/1180892451 wait complete. 2023-11-15T05:03:18.204 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.201+0000 7f78d2586700 1 Processor -- start 2023-11-15T05:03:18.204 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.201+0000 7f78d2586700 1 -- start start 2023-11-15T05:03:18.204 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.201+0000 7f78d2586700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78cc0f5410 0x7f78cc109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:18.204 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.201+0000 7f78d2586700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc060220 con 0x7f78cc0f5410 2023-11-15T05:03:18.205 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.201+0000 7f78cbfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78cc0f5410 0x7f78cc109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:18.205 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.201+0000 7f78cbfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78cc0f5410 0x7f78cc109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:41242/0 (socket says 172.21.15.178:41242) 2023-11-15T05:03:18.205 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.201+0000 7f78cbfff700 1 -- 172.21.15.178:0/3540602512 learned_addr learned my addr 172.21.15.178:0/3540602512 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:03:18.207 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.201+0000 7f78cbfff700 1 -- 172.21.15.178:0/3540602512 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b4005040 con 0x7f78cc0f5410 2023-11-15T05:03:18.207 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.201+0000 7f78cbfff700 1 --2- 172.21.15.178:0/3540602512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78cc0f5410 0x7f78cc109420 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f78b4013510 tx=0x7f78b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:18.208 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.202+0000 7f78c8ff9700 1 -- 172.21.15.178:0/3540602512 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78b4009320 con 0x7f78cc0f5410 2023-11-15T05:03:18.208 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.202+0000 7f78d2586700 1 -- 172.21.15.178:0/3540602512 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78cc060420 con 0x7f78cc0f5410 2023-11-15T05:03:18.208 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.202+0000 7f78c8ff9700 1 -- 172.21.15.178:0/3540602512 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f78b4013f00 con 0x7f78cc0f5410 2023-11-15T05:03:18.209 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.202+0000 7f78c8ff9700 1 -- 172.21.15.178:0/3540602512 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78b4014d90 con 0x7f78cc0f5410 2023-11-15T05:03:18.209 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.202+0000 7f78d2586700 1 -- 172.21.15.178:0/3540602512 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78cc109ad0 con 0x7f78cc0f5410 2023-11-15T05:03:18.209 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.202+0000 7f78c8ff9700 1 -- 172.21.15.178:0/3540602512 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f78b4014420 con 0x7f78cc0f5410 2023-11-15T05:03:18.209 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.202+0000 7f78d2586700 1 -- 172.21.15.178:0/3540602512 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78cc059100 con 0x7f78cc0f5410 2023-11-15T05:03:18.210 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.203+0000 7f78c8ff9700 1 --2- 172.21.15.178:0/3540602512 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f78b802ff60 0x7f78b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:18.210 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.203+0000 7f78c8ff9700 1 -- 172.21.15.178:0/3540602512 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f78b4041000 con 0x7f78cc0f5410 2023-11-15T05:03:18.210 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.203+0000 7f78cb7fe700 1 --2- 172.21.15.178:0/3540602512 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f78b802ff60 0x7f78b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:18.210 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.203+0000 7f78cb7fe700 1 --2- 172.21.15.178:0/3540602512 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f78b802ff60 0x7f78b8032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f78bc0096a0 tx=0x7f78bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:18.211 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.208+0000 7f78c8ff9700 1 -- 172.21.15.178:0/3540602512 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f78b40095f0 con 0x7f78cc0f5410 2023-11-15T05:03:18.412 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.411+0000 7f78d2586700 1 -- 172.21.15.178:0/3540602512 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f78cc045ad0 con 0x7f78cc0f5410 2023-11-15T05:03:18.413 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.413+0000 7f78c8ff9700 1 -- 172.21.15.178:0/3540602512 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f78b401db70 con 0x7f78cc0f5410 2023-11-15T05:03:18.413 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:03:18.414 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:18.415 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.413+0000 7f78d2586700 1 -- 172.21.15.178:0/3540602512 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f78b802ff60 msgr2=0x7f78b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:18.415 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.413+0000 7f78d2586700 1 --2- 172.21.15.178:0/3540602512 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f78b802ff60 0x7f78b8032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f78bc0096a0 tx=0x7f78bc006b40).stop 2023-11-15T05:03:18.415 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.413+0000 7f78d2586700 1 -- 172.21.15.178:0/3540602512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78cc0f5410 msgr2=0x7f78cc109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:18.415 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.413+0000 7f78d2586700 1 --2- 172.21.15.178:0/3540602512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78cc0f5410 0x7f78cc109420 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f78b4013510 tx=0x7f78b400b040).stop 2023-11-15T05:03:18.415 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.413+0000 7f78d2586700 1 -- 172.21.15.178:0/3540602512 shutdown_connections 2023-11-15T05:03:18.416 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.413+0000 7f78d2586700 1 --2- 172.21.15.178:0/3540602512 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f78b802ff60 0x7f78b8032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:18.416 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.413+0000 7f78d2586700 1 --2- 172.21.15.178:0/3540602512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78cc0f5410 0x7f78cc109420 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:18.416 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.413+0000 7f78d2586700 1 -- 172.21.15.178:0/3540602512 >> 172.21.15.178:0/3540602512 conn(0x7f78cc0f0b70 msgr2=0x7f78cc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:18.416 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.413+0000 7f78d2586700 1 -- 172.21.15.178:0/3540602512 shutdown_connections 2023-11-15T05:03:18.416 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:18.413+0000 7f78d2586700 1 -- 172.21.15.178:0/3540602512 wait complete. 2023-11-15T05:03:18.417 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:03:18.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:18 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:16.383180+0000 mgr.smithi163.qesyrx (mgr.14162) 242 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:19.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:19 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:18.413504+0000 mon.smithi163 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.178:0/3540602512' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:19.789 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:19.789 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:20.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:20 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:18.383606+0000 mgr.smithi163.qesyrx (mgr.14162) 243 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:21.014 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.013+0000 7fd2ca8a6700 1 -- 172.21.15.178:0/2269934057 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2c40f51a0 msgr2=0x7fd2c40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:21.015 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.013+0000 7fd2ca8a6700 1 --2- 172.21.15.178:0/2269934057 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2c40f51a0 0x7fd2c40f55c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fd2c0009fb0 tx=0x7fd2c0009b30).stop 2023-11-15T05:03:21.015 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.013+0000 7fd2ca8a6700 1 -- 172.21.15.178:0/2269934057 shutdown_connections 2023-11-15T05:03:21.015 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.013+0000 7fd2ca8a6700 1 --2- 172.21.15.178:0/2269934057 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2c40f51a0 0x7fd2c40f55c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:21.015 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.013+0000 7fd2ca8a6700 1 -- 172.21.15.178:0/2269934057 >> 172.21.15.178:0/2269934057 conn(0x7fd2c40f09a0 msgr2=0x7fd2c40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:21.016 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.013+0000 7fd2ca8a6700 1 -- 172.21.15.178:0/2269934057 shutdown_connections 2023-11-15T05:03:21.016 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.013+0000 7fd2ca8a6700 1 -- 172.21.15.178:0/2269934057 wait complete. 2023-11-15T05:03:21.016 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.013+0000 7fd2ca8a6700 1 Processor -- start 2023-11-15T05:03:21.016 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.013+0000 7fd2ca8a6700 1 -- start start 2023-11-15T05:03:21.018 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.014+0000 7fd2ca8a6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2c40f51a0 0x7fd2c41053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:21.018 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.014+0000 7fd2ca8a6700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c41058a0 con 0x7fd2c40f51a0 2023-11-15T05:03:21.018 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.014+0000 7fd2c98a4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2c40f51a0 0x7fd2c41053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:21.019 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.014+0000 7fd2c98a4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2c40f51a0 0x7fd2c41053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:41256/0 (socket says 172.21.15.178:41256) 2023-11-15T05:03:21.019 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.014+0000 7fd2c98a4700 1 -- 172.21.15.178:0/4176663930 learned_addr learned my addr 172.21.15.178:0/4176663930 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:03:21.019 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.014+0000 7fd2c98a4700 1 -- 172.21.15.178:0/4176663930 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c0005040 con 0x7fd2c40f51a0 2023-11-15T05:03:21.019 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.014+0000 7fd2c98a4700 1 --2- 172.21.15.178:0/4176663930 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2c40f51a0 0x7fd2c41053b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fd2c0013510 tx=0x7fd2c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:21.020 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.015+0000 7fd2ba7fc700 1 -- 172.21.15.178:0/4176663930 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2c0009320 con 0x7fd2c40f51a0 2023-11-15T05:03:21.020 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.015+0000 7fd2ca8a6700 1 -- 172.21.15.178:0/4176663930 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c4105aa0 con 0x7fd2c40f51a0 2023-11-15T05:03:21.020 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.015+0000 7fd2ba7fc700 1 -- 172.21.15.178:0/4176663930 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd2c0013f00 con 0x7fd2c40f51a0 2023-11-15T05:03:21.020 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.015+0000 7fd2ba7fc700 1 -- 172.21.15.178:0/4176663930 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2c0014dc0 con 0x7fd2c40f51a0 2023-11-15T05:03:21.021 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.015+0000 7fd2ca8a6700 1 -- 172.21.15.178:0/4176663930 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c4105f30 con 0x7fd2c40f51a0 2023-11-15T05:03:21.021 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.016+0000 7fd2ba7fc700 1 -- 172.21.15.178:0/4176663930 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd2c00095f0 con 0x7fd2c40f51a0 2023-11-15T05:03:21.021 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.016+0000 7fd2ca8a6700 1 -- 172.21.15.178:0/4176663930 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c4045ad0 con 0x7fd2c40f51a0 2023-11-15T05:03:21.022 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.016+0000 7fd2ba7fc700 1 --2- 172.21.15.178:0/4176663930 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd2b002ff60 0x7fd2b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:21.022 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.016+0000 7fd2ba7fc700 1 -- 172.21.15.178:0/4176663930 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd2c0041000 con 0x7fd2c40f51a0 2023-11-15T05:03:21.022 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.016+0000 7fd2c90a3700 1 --2- 172.21.15.178:0/4176663930 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd2b002ff60 0x7fd2b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:21.023 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.016+0000 7fd2c90a3700 1 --2- 172.21.15.178:0/4176663930 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd2b002ff60 0x7fd2b0032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fd2b40096a0 tx=0x7fd2b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:21.023 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.021+0000 7fd2ba7fc700 1 -- 172.21.15.178:0/4176663930 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd2c001c790 con 0x7fd2c40f51a0 2023-11-15T05:03:21.226 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.225+0000 7fd2ca8a6700 1 -- 172.21.15.178:0/4176663930 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd2c4106de0 con 0x7fd2c40f51a0 2023-11-15T05:03:21.227 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.225+0000 7fd2ba7fc700 1 -- 172.21.15.178:0/4176663930 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd2c0010230 con 0x7fd2c40f51a0 2023-11-15T05:03:21.227 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:03:21.227 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:21.228 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.227+0000 7fd2ca8a6700 1 -- 172.21.15.178:0/4176663930 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd2b002ff60 msgr2=0x7fd2b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:21.228 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.227+0000 7fd2ca8a6700 1 --2- 172.21.15.178:0/4176663930 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd2b002ff60 0x7fd2b0032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fd2b40096a0 tx=0x7fd2b4006b40).stop 2023-11-15T05:03:21.229 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.227+0000 7fd2ca8a6700 1 -- 172.21.15.178:0/4176663930 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2c40f51a0 msgr2=0x7fd2c41053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:21.229 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.227+0000 7fd2ca8a6700 1 --2- 172.21.15.178:0/4176663930 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2c40f51a0 0x7fd2c41053b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fd2c0013510 tx=0x7fd2c000b040).stop 2023-11-15T05:03:21.229 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.227+0000 7fd2ca8a6700 1 -- 172.21.15.178:0/4176663930 shutdown_connections 2023-11-15T05:03:21.229 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.227+0000 7fd2ca8a6700 1 --2- 172.21.15.178:0/4176663930 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fd2b002ff60 0x7fd2b0032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:21.229 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.227+0000 7fd2ca8a6700 1 --2- 172.21.15.178:0/4176663930 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2c40f51a0 0x7fd2c41053b0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:21.230 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.227+0000 7fd2ca8a6700 1 -- 172.21.15.178:0/4176663930 >> 172.21.15.178:0/4176663930 conn(0x7fd2c40f09a0 msgr2=0x7fd2c40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:21.230 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.227+0000 7fd2ca8a6700 1 -- 172.21.15.178:0/4176663930 shutdown_connections 2023-11-15T05:03:21.230 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:21.227+0000 7fd2ca8a6700 1 -- 172.21.15.178:0/4176663930 wait complete. 2023-11-15T05:03:21.230 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:03:22.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:22 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:20.383969+0000 mgr.smithi163.qesyrx (mgr.14162) 244 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:22.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:22 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:21.227250+0000 mon.smithi163 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.178:0/4176663930' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:22.624 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:22.624 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:23.867 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.864+0000 7fbcbeb02700 1 -- 172.21.15.178:0/185713447 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbcb80f6510 msgr2=0x7fbcb80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:23.868 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.864+0000 7fbcbeb02700 1 --2- 172.21.15.178:0/185713447 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbcb80f6510 0x7fbcb80f6930 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fbcac009fb0 tx=0x7fbcac009b30).stop 2023-11-15T05:03:23.868 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.864+0000 7fbcbeb02700 1 -- 172.21.15.178:0/185713447 shutdown_connections 2023-11-15T05:03:23.868 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.864+0000 7fbcbeb02700 1 --2- 172.21.15.178:0/185713447 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbcb80f6510 0x7fbcb80f6930 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:23.868 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.864+0000 7fbcbeb02700 1 -- 172.21.15.178:0/185713447 >> 172.21.15.178:0/185713447 conn(0x7fbcb80f1c70 msgr2=0x7fbcb80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:23.869 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.864+0000 7fbcbeb02700 1 -- 172.21.15.178:0/185713447 shutdown_connections 2023-11-15T05:03:23.869 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.864+0000 7fbcbeb02700 1 -- 172.21.15.178:0/185713447 wait complete. 2023-11-15T05:03:23.869 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.865+0000 7fbcbeb02700 1 Processor -- start 2023-11-15T05:03:23.869 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.865+0000 7fbcbeb02700 1 -- start start 2023-11-15T05:03:23.869 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.865+0000 7fbcbeb02700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbcb80f6510 0x7fbcb806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:23.870 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.865+0000 7fbcbeb02700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb8069e20 con 0x7fbcb80f6510 2023-11-15T05:03:23.870 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.865+0000 7fbcbc89e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbcb80f6510 0x7fbcb806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:23.870 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.865+0000 7fbcbc89e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbcb80f6510 0x7fbcb806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:41270/0 (socket says 172.21.15.178:41270) 2023-11-15T05:03:23.870 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.865+0000 7fbcbc89e700 1 -- 172.21.15.178:0/3383820201 learned_addr learned my addr 172.21.15.178:0/3383820201 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:03:23.870 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.866+0000 7fbcbc89e700 1 -- 172.21.15.178:0/3383820201 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcac005040 con 0x7fbcb80f6510 2023-11-15T05:03:23.871 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.866+0000 7fbcbc89e700 1 --2- 172.21.15.178:0/3383820201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbcb80f6510 0x7fbcb806b9a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fbcac013510 tx=0x7fbcac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:23.871 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.866+0000 7fbcb57fa700 1 -- 172.21.15.178:0/3383820201 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcac009320 con 0x7fbcb80f6510 2023-11-15T05:03:23.871 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.866+0000 7fbcb57fa700 1 -- 172.21.15.178:0/3383820201 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbcac013f00 con 0x7fbcb80f6510 2023-11-15T05:03:23.871 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.866+0000 7fbcbeb02700 1 -- 172.21.15.178:0/3383820201 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb8069fc0 con 0x7fbcb80f6510 2023-11-15T05:03:23.871 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.866+0000 7fbcb57fa700 1 -- 172.21.15.178:0/3383820201 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcac014d90 con 0x7fbcb80f6510 2023-11-15T05:03:23.872 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.866+0000 7fbcbeb02700 1 -- 172.21.15.178:0/3383820201 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb806a450 con 0x7fbcb80f6510 2023-11-15T05:03:23.872 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.867+0000 7fbcb57fa700 1 -- 172.21.15.178:0/3383820201 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbcac0095f0 con 0x7fbcb80f6510 2023-11-15T05:03:23.872 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.867+0000 7fbcbeb02700 1 -- 172.21.15.178:0/3383820201 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb8059100 con 0x7fbcb80f6510 2023-11-15T05:03:23.872 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.867+0000 7fbcb57fa700 1 --2- 172.21.15.178:0/3383820201 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fbca802ff60 0x7fbca8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:23.872 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.867+0000 7fbcb57fa700 1 -- 172.21.15.178:0/3383820201 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbcac0410c0 con 0x7fbcb80f6510 2023-11-15T05:03:23.873 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.867+0000 7fbcb7fff700 1 --2- 172.21.15.178:0/3383820201 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fbca802ff60 0x7fbca8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:23.873 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.867+0000 7fbcb7fff700 1 --2- 172.21.15.178:0/3383820201 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fbca802ff60 0x7fbca8032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fbca40096a0 tx=0x7fbca4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:23.874 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:23.872+0000 7fbcb57fa700 1 -- 172.21.15.178:0/3383820201 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbcac014420 con 0x7fbcb80f6510 2023-11-15T05:03:24.075 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:24.075+0000 7fbcbeb02700 1 -- 172.21.15.178:0/3383820201 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbcb80fbf60 con 0x7fbcb80f6510 2023-11-15T05:03:24.076 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:24.076+0000 7fbcb57fa700 1 -- 172.21.15.178:0/3383820201 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbcac010230 con 0x7fbcb80f6510 2023-11-15T05:03:24.076 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:03:24.077 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:24.078 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:24.077+0000 7fbcbeb02700 1 -- 172.21.15.178:0/3383820201 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fbca802ff60 msgr2=0x7fbca8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:24.078 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:24.077+0000 7fbcbeb02700 1 --2- 172.21.15.178:0/3383820201 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fbca802ff60 0x7fbca8032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fbca40096a0 tx=0x7fbca4006b40).stop 2023-11-15T05:03:24.078 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:24.077+0000 7fbcbeb02700 1 -- 172.21.15.178:0/3383820201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbcb80f6510 msgr2=0x7fbcb806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:24.078 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:24.077+0000 7fbcbeb02700 1 --2- 172.21.15.178:0/3383820201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbcb80f6510 0x7fbcb806b9a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fbcac013510 tx=0x7fbcac00b040).stop 2023-11-15T05:03:24.079 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:24.077+0000 7fbcbeb02700 1 -- 172.21.15.178:0/3383820201 shutdown_connections 2023-11-15T05:03:24.079 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:24.077+0000 7fbcbeb02700 1 --2- 172.21.15.178:0/3383820201 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fbca802ff60 0x7fbca8032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:24.079 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:24.077+0000 7fbcbeb02700 1 --2- 172.21.15.178:0/3383820201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbcb80f6510 0x7fbcb806b9a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:24.079 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:24.077+0000 7fbcbeb02700 1 -- 172.21.15.178:0/3383820201 >> 172.21.15.178:0/3383820201 conn(0x7fbcb80f1c70 msgr2=0x7fbcb80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:24.080 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:24.077+0000 7fbcbeb02700 1 -- 172.21.15.178:0/3383820201 shutdown_connections 2023-11-15T05:03:24.080 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:24.077+0000 7fbcbeb02700 1 -- 172.21.15.178:0/3383820201 wait complete. 2023-11-15T05:03:24.080 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:03:24.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:24 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:22.384328+0000 mgr.smithi163.qesyrx (mgr.14162) 245 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:24.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:24 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:24.076582+0000 mon.smithi163 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.178:0/3383820201' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:25.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:24.455919+0000 mon.smithi163 (mon.0) 300 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:03:25.453 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:24.456709+0000 mon.smithi163 (mon.0) 301 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/trash_purge_schedule"}]: dispatch 2023-11-15T05:03:25.469 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:25.470 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:26.452 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:26 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:24.384680+0000 mgr.smithi163.qesyrx (mgr.14162) 246 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:26.664 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.662+0000 7fdfa3b9f700 1 -- 172.21.15.178:0/3004019179 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdf9c0f5410 msgr2=0x7fdf9c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:26.664 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.662+0000 7fdfa3b9f700 1 --2- 172.21.15.178:0/3004019179 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdf9c0f5410 0x7fdf9c0f5830 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fdf8c009fb0 tx=0x7fdf8c009b30).stop 2023-11-15T05:03:26.664 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.662+0000 7fdfa3b9f700 1 -- 172.21.15.178:0/3004019179 shutdown_connections 2023-11-15T05:03:26.665 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.662+0000 7fdfa3b9f700 1 --2- 172.21.15.178:0/3004019179 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdf9c0f5410 0x7fdf9c0f5830 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:26.665 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.662+0000 7fdfa3b9f700 1 -- 172.21.15.178:0/3004019179 >> 172.21.15.178:0/3004019179 conn(0x7fdf9c0f0b70 msgr2=0x7fdf9c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:26.665 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.662+0000 7fdfa3b9f700 1 -- 172.21.15.178:0/3004019179 shutdown_connections 2023-11-15T05:03:26.665 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.662+0000 7fdfa3b9f700 1 -- 172.21.15.178:0/3004019179 wait complete. 2023-11-15T05:03:26.665 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.663+0000 7fdfa3b9f700 1 Processor -- start 2023-11-15T05:03:26.666 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.663+0000 7fdfa3b9f700 1 -- start start 2023-11-15T05:03:26.666 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.663+0000 7fdfa3b9f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdf9c0f5410 0x7fdf9c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:26.666 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.663+0000 7fdfa3b9f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c106c10 con 0x7fdf9c0f5410 2023-11-15T05:03:26.666 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.663+0000 7fdfa193b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdf9c0f5410 0x7fdf9c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:26.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.663+0000 7fdfa193b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdf9c0f5410 0x7fdf9c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:60026/0 (socket says 172.21.15.178:60026) 2023-11-15T05:03:26.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.663+0000 7fdfa193b700 1 -- 172.21.15.178:0/776397435 learned_addr learned my addr 172.21.15.178:0/776397435 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:03:26.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.663+0000 7fdfa193b700 1 -- 172.21.15.178:0/776397435 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf8c005040 con 0x7fdf9c0f5410 2023-11-15T05:03:26.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.663+0000 7fdfa193b700 1 --2- 172.21.15.178:0/776397435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdf9c0f5410 0x7fdf9c106720 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fdf8c013510 tx=0x7fdf8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:26.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.664+0000 7fdf927fc700 1 -- 172.21.15.178:0/776397435 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf8c009320 con 0x7fdf9c0f5410 2023-11-15T05:03:26.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.664+0000 7fdfa3b9f700 1 -- 172.21.15.178:0/776397435 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf9c106e10 con 0x7fdf9c0f5410 2023-11-15T05:03:26.668 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.664+0000 7fdf927fc700 1 -- 172.21.15.178:0/776397435 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf8c013f00 con 0x7fdf9c0f5410 2023-11-15T05:03:26.668 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.664+0000 7fdf927fc700 1 -- 172.21.15.178:0/776397435 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf8c014d90 con 0x7fdf9c0f5410 2023-11-15T05:03:26.668 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.664+0000 7fdfa3b9f700 1 -- 172.21.15.178:0/776397435 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf9c1072a0 con 0x7fdf9c0f5410 2023-11-15T05:03:26.669 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.664+0000 7fdf927fc700 1 -- 172.21.15.178:0/776397435 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdf8c0095f0 con 0x7fdf9c0f5410 2023-11-15T05:03:26.669 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.664+0000 7fdfa3b9f700 1 -- 172.21.15.178:0/776397435 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf9c045ad0 con 0x7fdf9c0f5410 2023-11-15T05:03:26.669 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.665+0000 7fdf927fc700 1 --2- 172.21.15.178:0/776397435 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdf8802ff60 0x7fdf88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:26.670 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.665+0000 7fdf927fc700 1 -- 172.21.15.178:0/776397435 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdf8c040fc0 con 0x7fdf9c0f5410 2023-11-15T05:03:26.670 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.665+0000 7fdfa113a700 1 --2- 172.21.15.178:0/776397435 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdf8802ff60 0x7fdf88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:26.670 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.665+0000 7fdfa113a700 1 --2- 172.21.15.178:0/776397435 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdf8802ff60 0x7fdf88032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fdf980096a0 tx=0x7fdf98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:26.671 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.669+0000 7fdf927fc700 1 -- 172.21.15.178:0/776397435 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdf8c014420 con 0x7fdf9c0f5410 2023-11-15T05:03:26.876 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.875+0000 7fdfa3b9f700 1 -- 172.21.15.178:0/776397435 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf9c108150 con 0x7fdf9c0f5410 2023-11-15T05:03:26.876 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.876+0000 7fdf927fc700 1 -- 172.21.15.178:0/776397435 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdf8c010230 con 0x7fdf9c0f5410 2023-11-15T05:03:26.877 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:03:26.877 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:26.878 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.878+0000 7fdfa3b9f700 1 -- 172.21.15.178:0/776397435 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdf8802ff60 msgr2=0x7fdf88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:26.878 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.878+0000 7fdfa3b9f700 1 --2- 172.21.15.178:0/776397435 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdf8802ff60 0x7fdf88032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fdf980096a0 tx=0x7fdf98006b40).stop 2023-11-15T05:03:26.878 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.878+0000 7fdfa3b9f700 1 -- 172.21.15.178:0/776397435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdf9c0f5410 msgr2=0x7fdf9c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:26.879 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.878+0000 7fdfa3b9f700 1 --2- 172.21.15.178:0/776397435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdf9c0f5410 0x7fdf9c106720 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fdf8c013510 tx=0x7fdf8c00b040).stop 2023-11-15T05:03:26.879 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.878+0000 7fdfa3b9f700 1 -- 172.21.15.178:0/776397435 shutdown_connections 2023-11-15T05:03:26.879 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.878+0000 7fdfa3b9f700 1 --2- 172.21.15.178:0/776397435 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fdf8802ff60 0x7fdf88032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:26.879 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.878+0000 7fdfa3b9f700 1 --2- 172.21.15.178:0/776397435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdf9c0f5410 0x7fdf9c106720 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:26.879 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.878+0000 7fdfa3b9f700 1 -- 172.21.15.178:0/776397435 >> 172.21.15.178:0/776397435 conn(0x7fdf9c0f0b70 msgr2=0x7fdf9c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:26.880 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.878+0000 7fdfa3b9f700 1 -- 172.21.15.178:0/776397435 shutdown_connections 2023-11-15T05:03:26.880 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:26.878+0000 7fdfa3b9f700 1 -- 172.21.15.178:0/776397435 wait complete. 2023-11-15T05:03:26.880 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:03:28.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:27 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:26.384983+0000 mgr.smithi163.qesyrx (mgr.14162) 247 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:28.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:27 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:26.876925+0000 mon.smithi163 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.178:0/776397435' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:28.269 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:28.270 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:29.453 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.450+0000 7f92ee8ec700 1 -- 172.21.15.178:0/2297280540 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92e80f5410 msgr2=0x7f92e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:29.453 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.450+0000 7f92ee8ec700 1 --2- 172.21.15.178:0/2297280540 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92e80f5410 0x7f92e80f5830 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f92dc009fb0 tx=0x7f92dc009b30).stop 2023-11-15T05:03:29.454 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.450+0000 7f92ee8ec700 1 -- 172.21.15.178:0/2297280540 shutdown_connections 2023-11-15T05:03:29.454 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.450+0000 7f92ee8ec700 1 --2- 172.21.15.178:0/2297280540 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92e80f5410 0x7f92e80f5830 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:29.454 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.450+0000 7f92ee8ec700 1 -- 172.21.15.178:0/2297280540 >> 172.21.15.178:0/2297280540 conn(0x7f92e80f0b70 msgr2=0x7f92e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:29.454 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.450+0000 7f92ee8ec700 1 -- 172.21.15.178:0/2297280540 shutdown_connections 2023-11-15T05:03:29.454 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.450+0000 7f92ee8ec700 1 -- 172.21.15.178:0/2297280540 wait complete. 2023-11-15T05:03:29.455 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.450+0000 7f92ee8ec700 1 Processor -- start 2023-11-15T05:03:29.455 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.451+0000 7f92ee8ec700 1 -- start start 2023-11-15T05:03:29.455 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.451+0000 7f92ee8ec700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92e80f5410 0x7f92e8106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:29.455 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.451+0000 7f92ee8ec700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e8106c10 con 0x7f92e80f5410 2023-11-15T05:03:29.456 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.451+0000 7f92e7fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92e80f5410 0x7f92e8106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:29.456 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.451+0000 7f92e7fff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92e80f5410 0x7f92e8106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:60042/0 (socket says 172.21.15.178:60042) 2023-11-15T05:03:29.456 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.451+0000 7f92e7fff700 1 -- 172.21.15.178:0/2191733431 learned_addr learned my addr 172.21.15.178:0/2191733431 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:03:29.457 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.451+0000 7f92e7fff700 1 -- 172.21.15.178:0/2191733431 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92dc005040 con 0x7f92e80f5410 2023-11-15T05:03:29.457 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.451+0000 7f92e7fff700 1 --2- 172.21.15.178:0/2191733431 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92e80f5410 0x7f92e8106720 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f92dc013510 tx=0x7f92dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:29.457 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.452+0000 7f92e4ff9700 1 -- 172.21.15.178:0/2191733431 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92dc009320 con 0x7f92e80f5410 2023-11-15T05:03:29.458 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.452+0000 7f92ee8ec700 1 -- 172.21.15.178:0/2191733431 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e8106e10 con 0x7f92e80f5410 2023-11-15T05:03:29.458 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.452+0000 7f92e4ff9700 1 -- 172.21.15.178:0/2191733431 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f92dc013f00 con 0x7f92e80f5410 2023-11-15T05:03:29.458 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.452+0000 7f92e4ff9700 1 -- 172.21.15.178:0/2191733431 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92dc014d90 con 0x7f92e80f5410 2023-11-15T05:03:29.459 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.452+0000 7f92ee8ec700 1 -- 172.21.15.178:0/2191733431 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e81072a0 con 0x7f92e80f5410 2023-11-15T05:03:29.459 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.452+0000 7f92e4ff9700 1 -- 172.21.15.178:0/2191733431 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f92dc014420 con 0x7f92e80f5410 2023-11-15T05:03:29.459 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.452+0000 7f92ee8ec700 1 -- 172.21.15.178:0/2191733431 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92e8045ad0 con 0x7f92e80f5410 2023-11-15T05:03:29.459 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.453+0000 7f92e4ff9700 1 --2- 172.21.15.178:0/2191733431 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f92d002ff60 0x7f92d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:29.459 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.453+0000 7f92e4ff9700 1 -- 172.21.15.178:0/2191733431 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f92dc041050 con 0x7f92e80f5410 2023-11-15T05:03:29.460 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.453+0000 7f92e77fe700 1 --2- 172.21.15.178:0/2191733431 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f92d002ff60 0x7f92d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:29.460 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.453+0000 7f92e77fe700 1 --2- 172.21.15.178:0/2191733431 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f92d002ff60 0x7f92d0032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f92d80096a0 tx=0x7f92d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:29.460 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.457+0000 7f92e4ff9700 1 -- 172.21.15.178:0/2191733431 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f92dc0095f0 con 0x7f92e80f5410 2023-11-15T05:03:29.663 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.663+0000 7f92ee8ec700 1 -- 172.21.15.178:0/2191733431 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f92e81080f0 con 0x7f92e80f5410 2023-11-15T05:03:29.663 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.663+0000 7f92e4ff9700 1 -- 172.21.15.178:0/2191733431 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f92dc01db70 con 0x7f92e80f5410 2023-11-15T05:03:29.664 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:03:29.664 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:29.666 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.665+0000 7f92ee8ec700 1 -- 172.21.15.178:0/2191733431 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f92d002ff60 msgr2=0x7f92d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:29.666 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.665+0000 7f92ee8ec700 1 --2- 172.21.15.178:0/2191733431 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f92d002ff60 0x7f92d0032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f92d80096a0 tx=0x7f92d8006b40).stop 2023-11-15T05:03:29.666 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.665+0000 7f92ee8ec700 1 -- 172.21.15.178:0/2191733431 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92e80f5410 msgr2=0x7f92e8106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:29.666 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.665+0000 7f92ee8ec700 1 --2- 172.21.15.178:0/2191733431 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92e80f5410 0x7f92e8106720 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f92dc013510 tx=0x7f92dc00b040).stop 2023-11-15T05:03:29.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.665+0000 7f92ee8ec700 1 -- 172.21.15.178:0/2191733431 shutdown_connections 2023-11-15T05:03:29.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.665+0000 7f92ee8ec700 1 --2- 172.21.15.178:0/2191733431 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f92d002ff60 0x7f92d0032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:29.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.665+0000 7f92ee8ec700 1 --2- 172.21.15.178:0/2191733431 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92e80f5410 0x7f92e8106720 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:29.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.665+0000 7f92ee8ec700 1 -- 172.21.15.178:0/2191733431 >> 172.21.15.178:0/2191733431 conn(0x7f92e80f0b70 msgr2=0x7f92e805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:29.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.665+0000 7f92ee8ec700 1 -- 172.21.15.178:0/2191733431 shutdown_connections 2023-11-15T05:03:29.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:29.665+0000 7f92ee8ec700 1 -- 172.21.15.178:0/2191733431 wait complete. 2023-11-15T05:03:29.668 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:03:30.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:29 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:28.385287+0000 mgr.smithi163.qesyrx (mgr.14162) 248 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:30.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:29 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:29.663934+0000 mon.smithi163 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.178:0/2191733431' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:31.079 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:31.080 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:32.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:31 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:30.385599+0000 mgr.smithi163.qesyrx (mgr.14162) 249 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:32.270 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.269+0000 7f4d8ca41700 1 -- 172.21.15.178:0/880768237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d880f62a0 msgr2=0x7f4d880f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:32.270 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.269+0000 7f4d8ca41700 1 --2- 172.21.15.178:0/880768237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d880f62a0 0x7f4d880f66c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f4d78009fb0 tx=0x7f4d78009b30).stop 2023-11-15T05:03:32.270 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.269+0000 7f4d8ca41700 1 -- 172.21.15.178:0/880768237 shutdown_connections 2023-11-15T05:03:32.271 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.269+0000 7f4d8ca41700 1 --2- 172.21.15.178:0/880768237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d880f62a0 0x7f4d880f66c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:32.271 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.269+0000 7f4d8ca41700 1 -- 172.21.15.178:0/880768237 >> 172.21.15.178:0/880768237 conn(0x7f4d880f1aa0 msgr2=0x7f4d880f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:32.271 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.269+0000 7f4d8ca41700 1 -- 172.21.15.178:0/880768237 shutdown_connections 2023-11-15T05:03:32.271 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.269+0000 7f4d8ca41700 1 -- 172.21.15.178:0/880768237 wait complete. 2023-11-15T05:03:32.271 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.269+0000 7f4d8ca41700 1 Processor -- start 2023-11-15T05:03:32.271 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.270+0000 7f4d8ca41700 1 -- start start 2023-11-15T05:03:32.272 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.270+0000 7f4d8ca41700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d880f62a0 0x7f4d881064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:32.272 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.270+0000 7f4d8ca41700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d881069a0 con 0x7f4d880f62a0 2023-11-15T05:03:32.274 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.270+0000 7f4d877fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d880f62a0 0x7f4d881064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:32.274 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.270+0000 7f4d877fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d880f62a0 0x7f4d881064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:60058/0 (socket says 172.21.15.178:60058) 2023-11-15T05:03:32.274 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.270+0000 7f4d877fe700 1 -- 172.21.15.178:0/4169303361 learned_addr learned my addr 172.21.15.178:0/4169303361 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:03:32.274 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.270+0000 7f4d877fe700 1 -- 172.21.15.178:0/4169303361 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d78005040 con 0x7f4d880f62a0 2023-11-15T05:03:32.274 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.270+0000 7f4d877fe700 1 --2- 172.21.15.178:0/4169303361 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d880f62a0 0x7f4d881064b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f4d78013510 tx=0x7f4d7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:32.275 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.270+0000 7f4d6ffff700 1 -- 172.21.15.178:0/4169303361 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d78009320 con 0x7f4d880f62a0 2023-11-15T05:03:32.275 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.271+0000 7f4d8ca41700 1 -- 172.21.15.178:0/4169303361 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d88106ba0 con 0x7f4d880f62a0 2023-11-15T05:03:32.275 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.271+0000 7f4d6ffff700 1 -- 172.21.15.178:0/4169303361 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d78013f00 con 0x7f4d880f62a0 2023-11-15T05:03:32.276 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.271+0000 7f4d6ffff700 1 -- 172.21.15.178:0/4169303361 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d78014dc0 con 0x7f4d880f62a0 2023-11-15T05:03:32.276 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.271+0000 7f4d8ca41700 1 -- 172.21.15.178:0/4169303361 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d88107030 con 0x7f4d880f62a0 2023-11-15T05:03:32.276 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.271+0000 7f4d6ffff700 1 -- 172.21.15.178:0/4169303361 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d7801d3d0 con 0x7f4d880f62a0 2023-11-15T05:03:32.276 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.272+0000 7f4d8ca41700 1 -- 172.21.15.178:0/4169303361 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d88045ad0 con 0x7f4d880f62a0 2023-11-15T05:03:32.276 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.272+0000 7f4d6ffff700 1 --2- 172.21.15.178:0/4169303361 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4d7002ff60 0x7f4d70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:32.277 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.272+0000 7f4d6ffff700 1 -- 172.21.15.178:0/4169303361 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d78041040 con 0x7f4d880f62a0 2023-11-15T05:03:32.277 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.272+0000 7f4d86ffd700 1 --2- 172.21.15.178:0/4169303361 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4d7002ff60 0x7f4d70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:32.277 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.272+0000 7f4d86ffd700 1 --2- 172.21.15.178:0/4169303361 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4d7002ff60 0x7f4d70032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f4d7c0096a0 tx=0x7f4d7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:32.277 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.276+0000 7f4d6ffff700 1 -- 172.21.15.178:0/4169303361 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d780126f0 con 0x7f4d880f62a0 2023-11-15T05:03:32.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.481+0000 7f4d8ca41700 1 -- 172.21.15.178:0/4169303361 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4d88107ee0 con 0x7f4d880f62a0 2023-11-15T05:03:32.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.482+0000 7f4d6ffff700 1 -- 172.21.15.178:0/4169303361 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4d7801ac30 con 0x7f4d880f62a0 2023-11-15T05:03:32.483 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:03:32.483 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:32.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.483+0000 7f4d8ca41700 1 -- 172.21.15.178:0/4169303361 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4d7002ff60 msgr2=0x7f4d70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:32.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.483+0000 7f4d8ca41700 1 --2- 172.21.15.178:0/4169303361 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4d7002ff60 0x7f4d70032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f4d7c0096a0 tx=0x7f4d7c006b40).stop 2023-11-15T05:03:32.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.483+0000 7f4d8ca41700 1 -- 172.21.15.178:0/4169303361 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d880f62a0 msgr2=0x7f4d881064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:32.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.483+0000 7f4d8ca41700 1 --2- 172.21.15.178:0/4169303361 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d880f62a0 0x7f4d881064b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f4d78013510 tx=0x7f4d7800b040).stop 2023-11-15T05:03:32.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.483+0000 7f4d8ca41700 1 -- 172.21.15.178:0/4169303361 shutdown_connections 2023-11-15T05:03:32.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.483+0000 7f4d8ca41700 1 --2- 172.21.15.178:0/4169303361 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4d7002ff60 0x7f4d70032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:32.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.483+0000 7f4d8ca41700 1 --2- 172.21.15.178:0/4169303361 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d880f62a0 0x7f4d881064b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:32.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.483+0000 7f4d8ca41700 1 -- 172.21.15.178:0/4169303361 >> 172.21.15.178:0/4169303361 conn(0x7f4d880f1aa0 msgr2=0x7f4d880f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:32.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.483+0000 7f4d8ca41700 1 -- 172.21.15.178:0/4169303361 shutdown_connections 2023-11-15T05:03:32.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:32.483+0000 7f4d8ca41700 1 -- 172.21.15.178:0/4169303361 wait complete. 2023-11-15T05:03:32.486 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:03:33.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:32 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:32.482376+0000 mon.smithi163 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.178:0/4169303361' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:33.875 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:33.876 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:34.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:33 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:32.385893+0000 mgr.smithi163.qesyrx (mgr.14162) 250 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:35.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.104+0000 7fcadcfd8700 1 -- 172.21.15.178:0/488349657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcad80f5410 msgr2=0x7fcad80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:35.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.104+0000 7fcadcfd8700 1 --2- 172.21.15.178:0/488349657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcad80f5410 0x7fcad80f5830 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fcac8009fb0 tx=0x7fcac8009b30).stop 2023-11-15T05:03:35.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.104+0000 7fcadcfd8700 1 -- 172.21.15.178:0/488349657 shutdown_connections 2023-11-15T05:03:35.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.104+0000 7fcadcfd8700 1 --2- 172.21.15.178:0/488349657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcad80f5410 0x7fcad80f5830 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:35.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.104+0000 7fcadcfd8700 1 -- 172.21.15.178:0/488349657 >> 172.21.15.178:0/488349657 conn(0x7fcad80f0b70 msgr2=0x7fcad80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:35.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.104+0000 7fcadcfd8700 1 -- 172.21.15.178:0/488349657 shutdown_connections 2023-11-15T05:03:35.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.104+0000 7fcadcfd8700 1 -- 172.21.15.178:0/488349657 wait complete. 2023-11-15T05:03:35.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.104+0000 7fcadcfd8700 1 Processor -- start 2023-11-15T05:03:35.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.104+0000 7fcadcfd8700 1 -- start start 2023-11-15T05:03:35.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.104+0000 7fcadcfd8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcad80f5410 0x7fcad8109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:35.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.105+0000 7fcadcfd8700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad8060240 con 0x7fcad80f5410 2023-11-15T05:03:35.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.105+0000 7fcad659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcad80f5410 0x7fcad8109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:35.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.105+0000 7fcad659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcad80f5410 0x7fcad8109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:57744/0 (socket says 172.21.15.178:57744) 2023-11-15T05:03:35.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.105+0000 7fcad659c700 1 -- 172.21.15.178:0/784402221 learned_addr learned my addr 172.21.15.178:0/784402221 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:03:35.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.105+0000 7fcad659c700 1 -- 172.21.15.178:0/784402221 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac8005040 con 0x7fcad80f5410 2023-11-15T05:03:35.109 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.105+0000 7fcad659c700 1 --2- 172.21.15.178:0/784402221 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcad80f5410 0x7fcad8109400 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fcac8013510 tx=0x7fcac800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:35.109 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.105+0000 7fcac6ffd700 1 -- 172.21.15.178:0/784402221 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcac8009320 con 0x7fcad80f5410 2023-11-15T05:03:35.109 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.105+0000 7fcadcfd8700 1 -- 172.21.15.178:0/784402221 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad8060440 con 0x7fcad80f5410 2023-11-15T05:03:35.109 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.105+0000 7fcac6ffd700 1 -- 172.21.15.178:0/784402221 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcac8013f00 con 0x7fcad80f5410 2023-11-15T05:03:35.109 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.105+0000 7fcac6ffd700 1 -- 172.21.15.178:0/784402221 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcac8014d90 con 0x7fcad80f5410 2023-11-15T05:03:35.110 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.105+0000 7fcadcfd8700 1 -- 172.21.15.178:0/784402221 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad8109ab0 con 0x7fcad80f5410 2023-11-15T05:03:35.111 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.106+0000 7fcac6ffd700 1 -- 172.21.15.178:0/784402221 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcac8014420 con 0x7fcad80f5410 2023-11-15T05:03:35.111 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.106+0000 7fcadcfd8700 1 -- 172.21.15.178:0/784402221 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad8045ad0 con 0x7fcad80f5410 2023-11-15T05:03:35.112 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.106+0000 7fcac6ffd700 1 --2- 172.21.15.178:0/784402221 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcac002ff60 0x7fcac0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:35.112 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.106+0000 7fcac6ffd700 1 -- 172.21.15.178:0/784402221 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcac80410c0 con 0x7fcad80f5410 2023-11-15T05:03:35.112 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.107+0000 7fcad5d9b700 1 --2- 172.21.15.178:0/784402221 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcac002ff60 0x7fcac0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:35.112 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.107+0000 7fcad5d9b700 1 --2- 172.21.15.178:0/784402221 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcac002ff60 0x7fcac0032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fcacc0096a0 tx=0x7fcacc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:35.112 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.111+0000 7fcac6ffd700 1 -- 172.21.15.178:0/784402221 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcac80095f0 con 0x7fcad80f5410 2023-11-15T05:03:35.316 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.316+0000 7fcadcfd8700 1 -- 172.21.15.178:0/784402221 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcad80fae60 con 0x7fcad80f5410 2023-11-15T05:03:35.316 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.316+0000 7fcac6ffd700 1 -- 172.21.15.178:0/784402221 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcac801db70 con 0x7fcad80f5410 2023-11-15T05:03:35.317 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:03:35.317 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:35.318 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.318+0000 7fcadcfd8700 1 -- 172.21.15.178:0/784402221 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcac002ff60 msgr2=0x7fcac0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:35.318 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.318+0000 7fcadcfd8700 1 --2- 172.21.15.178:0/784402221 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcac002ff60 0x7fcac0032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fcacc0096a0 tx=0x7fcacc006b40).stop 2023-11-15T05:03:35.319 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.318+0000 7fcadcfd8700 1 -- 172.21.15.178:0/784402221 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcad80f5410 msgr2=0x7fcad8109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:35.319 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.318+0000 7fcadcfd8700 1 --2- 172.21.15.178:0/784402221 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcad80f5410 0x7fcad8109400 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fcac8013510 tx=0x7fcac800b040).stop 2023-11-15T05:03:35.319 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.318+0000 7fcadcfd8700 1 -- 172.21.15.178:0/784402221 shutdown_connections 2023-11-15T05:03:35.319 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.318+0000 7fcadcfd8700 1 --2- 172.21.15.178:0/784402221 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fcac002ff60 0x7fcac0032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:35.319 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.318+0000 7fcadcfd8700 1 --2- 172.21.15.178:0/784402221 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcad80f5410 0x7fcad8109400 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:35.320 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.318+0000 7fcadcfd8700 1 -- 172.21.15.178:0/784402221 >> 172.21.15.178:0/784402221 conn(0x7fcad80f0b70 msgr2=0x7fcad80f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:35.320 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.318+0000 7fcadcfd8700 1 -- 172.21.15.178:0/784402221 shutdown_connections 2023-11-15T05:03:35.320 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:35.318+0000 7fcadcfd8700 1 -- 172.21.15.178:0/784402221 wait complete. 2023-11-15T05:03:35.320 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:03:36.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:34.386141+0000 mgr.smithi163.qesyrx (mgr.14162) 251 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:36.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cephadm 2023-11-15T05:03:34.493131+0000 mgr.smithi163.qesyrx (mgr.14162) 252 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi163 ... 2023-11-15T05:03:36.204 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Verifying port 9100 ... 2023-11-15T05:03:36.204 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Non-zero exit code 1 from systemctl start ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@node-exporter.smithi163 2023-11-15T05:03:36.204 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: systemctl: stderr Job for ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@node-exporter.smithi163.service failed because a timeout was exceeded. 2023-11-15T05:03:36.204 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: systemctl: stderr See "systemctl status ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@node-exporter.smithi163.service" and "journalctl -xe" for details. 2023-11-15T05:03:36.204 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Traceback (most recent call last): 2023-11-15T05:03:36.205 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T05:03:36.205 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: main() 2023-11-15T05:03:36.205 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T05:03:36.205 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: r = ctx.func(ctx) 2023-11-15T05:03:36.205 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T05:03:36.206 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: return func(ctx) 2023-11-15T05:03:36.206 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-15T05:03:36.206 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: ports=daemon_ports) 2023-11-15T05:03:36.206 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-15T05:03:36.206 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-15T05:03:36.207 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-15T05:03:36.207 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-15T05:03:36.207 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-15T05:03:36.208 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-15T05:03:36.208 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: RuntimeError: Failed command: systemctl start ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@node-exporter.smithi163 2023-11-15T05:03:36.208 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Traceback (most recent call last): 2023-11-15T05:03:36.209 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-15T05:03:36.209 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: yield (conn, connr) 2023-11-15T05:03:36.209 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-15T05:03:36.209 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: code, '\n'.join(err))) 2023-11-15T05:03:36.209 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi163 ... 2023-11-15T05:03:36.209 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Verifying port 9100 ... 2023-11-15T05:03:36.210 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Non-zero exit code 1 from systemctl start ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@node-exporter.smithi163 2023-11-15T05:03:36.210 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: systemctl: stderr Job for ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@node-exporter.smithi163.service failed because a timeout was exceeded. 2023-11-15T05:03:36.210 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: systemctl: stderr See "systemctl status ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@node-exporter.smithi163.service" and "journalctl -xe" for details. 2023-11-15T05:03:36.210 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: Traceback (most recent call last): 2023-11-15T05:03:36.210 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T05:03:36.211 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: main() 2023-11-15T05:03:36.211 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T05:03:36.211 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: r = ctx.func(ctx) 2023-11-15T05:03:36.211 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T05:03:36.211 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: return func(ctx) 2023-11-15T05:03:36.212 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-15T05:03:36.212 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: ports=daemon_ports) 2023-11-15T05:03:36.212 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-15T05:03:36.212 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-15T05:03:36.212 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-15T05:03:36.213 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-15T05:03:36.213 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: File "/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-15T05:03:36.213 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-15T05:03:36.213 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: RuntimeError: Failed command: systemctl start ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@node-exporter.smithi163 2023-11-15T05:03:36.213 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cephadm 2023-11-15T05:03:34.498854+0000 mgr.smithi163.qesyrx (mgr.14162) 253 : cephadm [INF] Deploying daemon prometheus.smithi163 on smithi163 2023-11-15T05:03:36.213 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:35 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:35.317215+0000 mon.smithi163 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.178:0/784402221' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:36.700 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:36.701 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:37.854 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.853+0000 7f2447f8d700 1 -- 172.21.15.178:0/1846835174 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24400f5410 msgr2=0x7f24400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:37.854 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.853+0000 7f2447f8d700 1 --2- 172.21.15.178:0/1846835174 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24400f5410 0x7f24400f5830 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f2434009fb0 tx=0x7f2434009b30).stop 2023-11-15T05:03:37.854 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.853+0000 7f2447f8d700 1 -- 172.21.15.178:0/1846835174 shutdown_connections 2023-11-15T05:03:37.855 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.853+0000 7f2447f8d700 1 --2- 172.21.15.178:0/1846835174 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24400f5410 0x7f24400f5830 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:37.855 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.853+0000 7f2447f8d700 1 -- 172.21.15.178:0/1846835174 >> 172.21.15.178:0/1846835174 conn(0x7f24400f0b70 msgr2=0x7f24400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:37.855 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.854+0000 7f2447f8d700 1 -- 172.21.15.178:0/1846835174 shutdown_connections 2023-11-15T05:03:37.855 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.854+0000 7f2447f8d700 1 -- 172.21.15.178:0/1846835174 wait complete. 2023-11-15T05:03:37.857 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.854+0000 7f2447f8d700 1 Processor -- start 2023-11-15T05:03:37.858 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.854+0000 7f2447f8d700 1 -- start start 2023-11-15T05:03:37.858 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.854+0000 7f2447f8d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24400f5410 0x7f2440109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:37.858 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.854+0000 7f2447f8d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2440060220 con 0x7f24400f5410 2023-11-15T05:03:37.859 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.854+0000 7f2445d29700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24400f5410 0x7f2440109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:37.859 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.854+0000 7f2445d29700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24400f5410 0x7f2440109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:57762/0 (socket says 172.21.15.178:57762) 2023-11-15T05:03:37.859 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.854+0000 7f2445d29700 1 -- 172.21.15.178:0/767318293 learned_addr learned my addr 172.21.15.178:0/767318293 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:03:37.859 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.855+0000 7f2445d29700 1 -- 172.21.15.178:0/767318293 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2434005040 con 0x7f24400f5410 2023-11-15T05:03:37.860 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.855+0000 7f2445d29700 1 --2- 172.21.15.178:0/767318293 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24400f5410 0x7f2440109450 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f2434014000 tx=0x7f243400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:37.860 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.855+0000 7f24327fc700 1 -- 172.21.15.178:0/767318293 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24340095f0 con 0x7f24400f5410 2023-11-15T05:03:37.860 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.855+0000 7f24327fc700 1 -- 172.21.15.178:0/767318293 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2434007e70 con 0x7f24400f5410 2023-11-15T05:03:37.860 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.855+0000 7f2447f8d700 1 -- 172.21.15.178:0/767318293 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2440060420 con 0x7f24400f5410 2023-11-15T05:03:37.860 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.855+0000 7f24327fc700 1 -- 172.21.15.178:0/767318293 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2434014ea0 con 0x7f24400f5410 2023-11-15T05:03:37.861 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.855+0000 7f2447f8d700 1 -- 172.21.15.178:0/767318293 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2440109b00 con 0x7f24400f5410 2023-11-15T05:03:37.861 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.856+0000 7f24327fc700 1 -- 172.21.15.178:0/767318293 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2434025070 con 0x7f24400f5410 2023-11-15T05:03:37.861 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.856+0000 7f2447f8d700 1 -- 172.21.15.178:0/767318293 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2440059100 con 0x7f24400f5410 2023-11-15T05:03:37.861 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.856+0000 7f24327fc700 1 --2- 172.21.15.178:0/767318293 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f242c02ff60 0x7f242c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:37.861 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.856+0000 7f24327fc700 1 -- 172.21.15.178:0/767318293 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2434040920 con 0x7f24400f5410 2023-11-15T05:03:37.862 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.856+0000 7f2445528700 1 --2- 172.21.15.178:0/767318293 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f242c02ff60 0x7f242c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:37.862 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.857+0000 7f2445528700 1 --2- 172.21.15.178:0/767318293 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f242c02ff60 0x7f242c032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f243c0096a0 tx=0x7f243c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:37.863 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:37.861+0000 7f24327fc700 1 -- 172.21.15.178:0/767318293 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f243400f4e0 con 0x7f24400f5410 2023-11-15T05:03:38.068 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:38.068+0000 7f2447f8d700 1 -- 172.21.15.178:0/767318293 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2440045ad0 con 0x7f24400f5410 2023-11-15T05:03:38.069 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:38.069+0000 7f24327fc700 1 -- 172.21.15.178:0/767318293 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f243400f300 con 0x7f24400f5410 2023-11-15T05:03:38.069 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:03:38.070 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:38.071 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:38.070+0000 7f2447f8d700 1 -- 172.21.15.178:0/767318293 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f242c02ff60 msgr2=0x7f242c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:38.071 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:38.070+0000 7f2447f8d700 1 --2- 172.21.15.178:0/767318293 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f242c02ff60 0x7f242c032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f243c0096a0 tx=0x7f243c006b40).stop 2023-11-15T05:03:38.071 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:38.070+0000 7f2447f8d700 1 -- 172.21.15.178:0/767318293 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24400f5410 msgr2=0x7f2440109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:38.071 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:38.070+0000 7f2447f8d700 1 --2- 172.21.15.178:0/767318293 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24400f5410 0x7f2440109450 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f2434014000 tx=0x7f243400b040).stop 2023-11-15T05:03:38.072 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:38.070+0000 7f2447f8d700 1 -- 172.21.15.178:0/767318293 shutdown_connections 2023-11-15T05:03:38.072 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:38.070+0000 7f2447f8d700 1 --2- 172.21.15.178:0/767318293 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f242c02ff60 0x7f242c032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:38.072 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:38.070+0000 7f2447f8d700 1 --2- 172.21.15.178:0/767318293 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24400f5410 0x7f2440109450 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:38.072 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:38.070+0000 7f2447f8d700 1 -- 172.21.15.178:0/767318293 >> 172.21.15.178:0/767318293 conn(0x7f24400f0b70 msgr2=0x7f24400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:38.072 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:38.070+0000 7f2447f8d700 1 -- 172.21.15.178:0/767318293 shutdown_connections 2023-11-15T05:03:38.072 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:38.070+0000 7f2447f8d700 1 -- 172.21.15.178:0/767318293 wait complete. 2023-11-15T05:03:38.073 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:03:38.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:37 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:36.386588+0000 mgr.smithi163.qesyrx (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:39.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:38 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:38.069573+0000 mon.smithi163 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.178:0/767318293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:39.496 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:39.497 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:40.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:39 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:38.386884+0000 mgr.smithi163.qesyrx (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:40.708 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.704+0000 7feb0265b700 1 -- 172.21.15.178:0/1189533602 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feafc0f6480 msgr2=0x7feafc0f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:40.708 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.704+0000 7feb0265b700 1 --2- 172.21.15.178:0/1189533602 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feafc0f6480 0x7feafc0f68a0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7feaec009fb0 tx=0x7feaec009b30).stop 2023-11-15T05:03:40.709 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.704+0000 7feb0265b700 1 -- 172.21.15.178:0/1189533602 shutdown_connections 2023-11-15T05:03:40.709 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.704+0000 7feb0265b700 1 --2- 172.21.15.178:0/1189533602 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feafc0f6480 0x7feafc0f68a0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:40.710 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.704+0000 7feb0265b700 1 -- 172.21.15.178:0/1189533602 >> 172.21.15.178:0/1189533602 conn(0x7feafc0f1c00 msgr2=0x7feafc0f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:40.710 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.704+0000 7feb0265b700 1 -- 172.21.15.178:0/1189533602 shutdown_connections 2023-11-15T05:03:40.710 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.704+0000 7feb0265b700 1 -- 172.21.15.178:0/1189533602 wait complete. 2023-11-15T05:03:40.711 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.705+0000 7feb0265b700 1 Processor -- start 2023-11-15T05:03:40.711 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.705+0000 7feb0265b700 1 -- start start 2023-11-15T05:03:40.711 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.705+0000 7feb0265b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feafc0f6480 0x7feafc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:40.712 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.705+0000 7feb0265b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feafc069e20 con 0x7feafc0f6480 2023-11-15T05:03:40.712 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.705+0000 7feafbfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feafc0f6480 0x7feafc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:40.712 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.705+0000 7feafbfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feafc0f6480 0x7feafc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:57778/0 (socket says 172.21.15.178:57778) 2023-11-15T05:03:40.713 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.705+0000 7feafbfff700 1 -- 172.21.15.178:0/96583518 learned_addr learned my addr 172.21.15.178:0/96583518 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:03:40.713 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.705+0000 7feafbfff700 1 -- 172.21.15.178:0/96583518 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaec005040 con 0x7feafc0f6480 2023-11-15T05:03:40.714 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.706+0000 7feafbfff700 1 --2- 172.21.15.178:0/96583518 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feafc0f6480 0x7feafc06b9a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7feaec013510 tx=0x7feaec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:40.714 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.706+0000 7feaf8ff9700 1 -- 172.21.15.178:0/96583518 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feaec009320 con 0x7feafc0f6480 2023-11-15T05:03:40.714 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.706+0000 7feb0265b700 1 -- 172.21.15.178:0/96583518 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feafc069fc0 con 0x7feafc0f6480 2023-11-15T05:03:40.715 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.706+0000 7feaf8ff9700 1 -- 172.21.15.178:0/96583518 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feaec013f00 con 0x7feafc0f6480 2023-11-15T05:03:40.715 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.706+0000 7feaf8ff9700 1 -- 172.21.15.178:0/96583518 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feaec014d90 con 0x7feafc0f6480 2023-11-15T05:03:40.715 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.706+0000 7feb0265b700 1 -- 172.21.15.178:0/96583518 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feafc06a450 con 0x7feafc0f6480 2023-11-15T05:03:40.716 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.707+0000 7feaf8ff9700 1 -- 172.21.15.178:0/96583518 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feaec0095f0 con 0x7feafc0f6480 2023-11-15T05:03:40.716 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.707+0000 7feb0265b700 1 -- 172.21.15.178:0/96583518 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feafc059100 con 0x7feafc0f6480 2023-11-15T05:03:40.717 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.707+0000 7feaf8ff9700 1 --2- 172.21.15.178:0/96583518 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feae402ff60 0x7feae4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:40.717 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.707+0000 7feaf8ff9700 1 -- 172.21.15.178:0/96583518 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feaec040fe0 con 0x7feafc0f6480 2023-11-15T05:03:40.717 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.707+0000 7feafb7fe700 1 --2- 172.21.15.178:0/96583518 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feae402ff60 0x7feae4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:40.718 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.707+0000 7feafb7fe700 1 --2- 172.21.15.178:0/96583518 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feae402ff60 0x7feae4032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7feaf00096a0 tx=0x7feaf0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:40.719 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.712+0000 7feaf8ff9700 1 -- 172.21.15.178:0/96583518 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feaec01d7f0 con 0x7feafc0f6480 2023-11-15T05:03:40.915 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.914+0000 7feb0265b700 1 -- 172.21.15.178:0/96583518 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feafc0fbf40 con 0x7feafc0f6480 2023-11-15T05:03:40.915 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.915+0000 7feaf8ff9700 1 -- 172.21.15.178:0/96583518 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7feaec010230 con 0x7feafc0f6480 2023-11-15T05:03:40.916 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:03:40.916 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:40.917 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.916+0000 7feb0265b700 1 -- 172.21.15.178:0/96583518 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feae402ff60 msgr2=0x7feae4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:40.917 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.916+0000 7feb0265b700 1 --2- 172.21.15.178:0/96583518 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feae402ff60 0x7feae4032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7feaf00096a0 tx=0x7feaf0006b40).stop 2023-11-15T05:03:40.917 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.916+0000 7feb0265b700 1 -- 172.21.15.178:0/96583518 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feafc0f6480 msgr2=0x7feafc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:40.918 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.916+0000 7feb0265b700 1 --2- 172.21.15.178:0/96583518 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feafc0f6480 0x7feafc06b9a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7feaec013510 tx=0x7feaec00b040).stop 2023-11-15T05:03:40.918 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.916+0000 7feb0265b700 1 -- 172.21.15.178:0/96583518 shutdown_connections 2023-11-15T05:03:40.918 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.916+0000 7feb0265b700 1 --2- 172.21.15.178:0/96583518 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7feae402ff60 0x7feae4032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:40.918 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.916+0000 7feb0265b700 1 --2- 172.21.15.178:0/96583518 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feafc0f6480 0x7feafc06b9a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:40.918 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.917+0000 7feb0265b700 1 -- 172.21.15.178:0/96583518 >> 172.21.15.178:0/96583518 conn(0x7feafc0f1c00 msgr2=0x7feafc0f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:40.919 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.917+0000 7feb0265b700 1 -- 172.21.15.178:0/96583518 shutdown_connections 2023-11-15T05:03:40.919 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:40.917+0000 7feb0265b700 1 -- 172.21.15.178:0/96583518 wait complete. 2023-11-15T05:03:40.919 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:03:42.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:41 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:40.387283+0000 mgr.smithi163.qesyrx (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:42.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:41 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:40.915944+0000 mon.smithi163 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.178:0/96583518' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:42.291 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:42.292 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:43.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.485+0000 7fe0956df700 1 -- 172.21.15.178:0/1045757542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe090060160 msgr2=0x7fe0900f8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:43.487 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.485+0000 7fe0956df700 1 --2- 172.21.15.178:0/1045757542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe090060160 0x7fe0900f8fa0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fe080009fb0 tx=0x7fe080009b30).stop 2023-11-15T05:03:43.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.485+0000 7fe0956df700 1 -- 172.21.15.178:0/1045757542 shutdown_connections 2023-11-15T05:03:43.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.485+0000 7fe0956df700 1 --2- 172.21.15.178:0/1045757542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe090060160 0x7fe0900f8fa0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:43.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.485+0000 7fe0956df700 1 -- 172.21.15.178:0/1045757542 >> 172.21.15.178:0/1045757542 conn(0x7fe0900f09a0 msgr2=0x7fe0900f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:43.488 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.485+0000 7fe0956df700 1 -- 172.21.15.178:0/1045757542 shutdown_connections 2023-11-15T05:03:43.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.485+0000 7fe0956df700 1 -- 172.21.15.178:0/1045757542 wait complete. 2023-11-15T05:03:43.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.485+0000 7fe0956df700 1 Processor -- start 2023-11-15T05:03:43.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.486+0000 7fe0956df700 1 -- start start 2023-11-15T05:03:43.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.486+0000 7fe0956df700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe090060160 0x7fe0900f6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:43.489 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.486+0000 7fe0956df700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0900f4ea0 con 0x7fe090060160 2023-11-15T05:03:43.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.486+0000 7fe08ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe090060160 0x7fe0900f6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:43.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.486+0000 7fe08ffff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe090060160 0x7fe0900f6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:57782/0 (socket says 172.21.15.178:57782) 2023-11-15T05:03:43.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.486+0000 7fe08ffff700 1 -- 172.21.15.178:0/1369937389 learned_addr learned my addr 172.21.15.178:0/1369937389 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:03:43.490 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.486+0000 7fe08ffff700 1 -- 172.21.15.178:0/1369937389 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe080005040 con 0x7fe090060160 2023-11-15T05:03:43.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.486+0000 7fe08ffff700 1 --2- 172.21.15.178:0/1369937389 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe090060160 0x7fe0900f6a20 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fe080013510 tx=0x7fe08000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:43.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.487+0000 7fe08cff9700 1 -- 172.21.15.178:0/1369937389 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe080009320 con 0x7fe090060160 2023-11-15T05:03:43.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.487+0000 7fe0956df700 1 -- 172.21.15.178:0/1369937389 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0900f5040 con 0x7fe090060160 2023-11-15T05:03:43.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.487+0000 7fe08cff9700 1 -- 172.21.15.178:0/1369937389 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe080013f00 con 0x7fe090060160 2023-11-15T05:03:43.491 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.487+0000 7fe08cff9700 1 -- 172.21.15.178:0/1369937389 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe080014d90 con 0x7fe090060160 2023-11-15T05:03:43.492 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.487+0000 7fe0956df700 1 -- 172.21.15.178:0/1369937389 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0900f54d0 con 0x7fe090060160 2023-11-15T05:03:43.493 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.488+0000 7fe0956df700 1 -- 172.21.15.178:0/1369937389 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe090059100 con 0x7fe090060160 2023-11-15T05:03:43.493 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.488+0000 7fe08cff9700 1 -- 172.21.15.178:0/1369937389 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0800095f0 con 0x7fe090060160 2023-11-15T05:03:43.493 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.488+0000 7fe08cff9700 1 --2- 172.21.15.178:0/1369937389 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe07c02ff60 0x7fe07c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:43.493 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.488+0000 7fe08cff9700 1 -- 172.21.15.178:0/1369937389 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe080040fc0 con 0x7fe090060160 2023-11-15T05:03:43.494 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.488+0000 7fe08f7fe700 1 --2- 172.21.15.178:0/1369937389 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe07c02ff60 0x7fe07c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:43.494 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.489+0000 7fe08f7fe700 1 --2- 172.21.15.178:0/1369937389 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe07c02ff60 0x7fe07c032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fe0780096a0 tx=0x7fe078006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:43.494 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.493+0000 7fe08cff9700 1 -- 172.21.15.178:0/1369937389 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe080014420 con 0x7fe090060160 2023-11-15T05:03:43.698 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.697+0000 7fe0956df700 1 -- 172.21.15.178:0/1369937389 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe090045ad0 con 0x7fe090060160 2023-11-15T05:03:43.698 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.698+0000 7fe08cff9700 1 -- 172.21.15.178:0/1369937389 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe080010230 con 0x7fe090060160 2023-11-15T05:03:43.699 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:03:43.699 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:43.700 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.699+0000 7fe0956df700 1 -- 172.21.15.178:0/1369937389 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe07c02ff60 msgr2=0x7fe07c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:43.700 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.699+0000 7fe0956df700 1 --2- 172.21.15.178:0/1369937389 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe07c02ff60 0x7fe07c032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fe0780096a0 tx=0x7fe078006b40).stop 2023-11-15T05:03:43.700 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.700+0000 7fe0956df700 1 -- 172.21.15.178:0/1369937389 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe090060160 msgr2=0x7fe0900f6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:43.700 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.700+0000 7fe0956df700 1 --2- 172.21.15.178:0/1369937389 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe090060160 0x7fe0900f6a20 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fe080013510 tx=0x7fe08000b040).stop 2023-11-15T05:03:43.701 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.700+0000 7fe0956df700 1 -- 172.21.15.178:0/1369937389 shutdown_connections 2023-11-15T05:03:43.701 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.700+0000 7fe0956df700 1 --2- 172.21.15.178:0/1369937389 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe07c02ff60 0x7fe07c032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:43.701 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.700+0000 7fe0956df700 1 --2- 172.21.15.178:0/1369937389 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe090060160 0x7fe0900f6a20 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:43.701 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.700+0000 7fe0956df700 1 -- 172.21.15.178:0/1369937389 >> 172.21.15.178:0/1369937389 conn(0x7fe0900f09a0 msgr2=0x7fe0900f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:43.701 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.700+0000 7fe0956df700 1 -- 172.21.15.178:0/1369937389 shutdown_connections 2023-11-15T05:03:43.702 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:43.700+0000 7fe0956df700 1 -- 172.21.15.178:0/1369937389 wait complete. 2023-11-15T05:03:43.702 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:03:44.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:43 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:42.387772+0000 mgr.smithi163.qesyrx (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:44.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:43 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:43.698795+0000 mon.smithi163 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.178:0/1369937389' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:45.098 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:45.098 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:46.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:45 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:44.388142+0000 mgr.smithi163.qesyrx (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:46.266 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.266+0000 7fca85f24700 1 -- 172.21.15.178:0/1421758965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca800f51a0 msgr2=0x7fca800f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:46.267 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.266+0000 7fca85f24700 1 --2- 172.21.15.178:0/1421758965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca800f51a0 0x7fca800f55c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fca74009fb0 tx=0x7fca74009b30).stop 2023-11-15T05:03:46.267 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.266+0000 7fca85f24700 1 -- 172.21.15.178:0/1421758965 shutdown_connections 2023-11-15T05:03:46.267 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.266+0000 7fca85f24700 1 --2- 172.21.15.178:0/1421758965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca800f51a0 0x7fca800f55c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:46.267 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.266+0000 7fca85f24700 1 -- 172.21.15.178:0/1421758965 >> 172.21.15.178:0/1421758965 conn(0x7fca800f09a0 msgr2=0x7fca800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:46.268 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.266+0000 7fca85f24700 1 -- 172.21.15.178:0/1421758965 shutdown_connections 2023-11-15T05:03:46.268 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.266+0000 7fca85f24700 1 -- 172.21.15.178:0/1421758965 wait complete. 2023-11-15T05:03:46.268 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.266+0000 7fca85f24700 1 Processor -- start 2023-11-15T05:03:46.268 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.266+0000 7fca85f24700 1 -- start start 2023-11-15T05:03:46.268 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.266+0000 7fca85f24700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca800f51a0 0x7fca801064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:46.269 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.266+0000 7fca85f24700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca80106990 con 0x7fca800f51a0 2023-11-15T05:03:46.270 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.267+0000 7fca84f22700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca800f51a0 0x7fca801064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:46.271 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.267+0000 7fca84f22700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca800f51a0 0x7fca801064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:32776/0 (socket says 172.21.15.178:32776) 2023-11-15T05:03:46.271 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.267+0000 7fca84f22700 1 -- 172.21.15.178:0/2693003471 learned_addr learned my addr 172.21.15.178:0/2693003471 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:03:46.271 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.267+0000 7fca84f22700 1 -- 172.21.15.178:0/2693003471 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca74005040 con 0x7fca800f51a0 2023-11-15T05:03:46.271 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.267+0000 7fca84f22700 1 --2- 172.21.15.178:0/2693003471 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca800f51a0 0x7fca801064a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fca74013510 tx=0x7fca7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:46.271 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.267+0000 7fca7d7fa700 1 -- 172.21.15.178:0/2693003471 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca74009320 con 0x7fca800f51a0 2023-11-15T05:03:46.272 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.267+0000 7fca85f24700 1 -- 172.21.15.178:0/2693003471 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca80106b90 con 0x7fca800f51a0 2023-11-15T05:03:46.272 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.267+0000 7fca7d7fa700 1 -- 172.21.15.178:0/2693003471 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca74013f00 con 0x7fca800f51a0 2023-11-15T05:03:46.272 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.267+0000 7fca7d7fa700 1 -- 172.21.15.178:0/2693003471 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca74014dc0 con 0x7fca800f51a0 2023-11-15T05:03:46.272 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.268+0000 7fca85f24700 1 -- 172.21.15.178:0/2693003471 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca80107020 con 0x7fca800f51a0 2023-11-15T05:03:46.272 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.268+0000 7fca7d7fa700 1 -- 172.21.15.178:0/2693003471 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca740095f0 con 0x7fca800f51a0 2023-11-15T05:03:46.273 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.268+0000 7fca85f24700 1 -- 172.21.15.178:0/2693003471 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca80045ad0 con 0x7fca800f51a0 2023-11-15T05:03:46.273 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.268+0000 7fca7d7fa700 1 --2- 172.21.15.178:0/2693003471 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fca7002ff60 0x7fca70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:46.273 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.269+0000 7fca7d7fa700 1 -- 172.21.15.178:0/2693003471 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca74041100 con 0x7fca800f51a0 2023-11-15T05:03:46.273 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.269+0000 7fca7ffff700 1 --2- 172.21.15.178:0/2693003471 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fca7002ff60 0x7fca70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:46.273 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.269+0000 7fca7ffff700 1 --2- 172.21.15.178:0/2693003471 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fca7002ff60 0x7fca70032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fca6c0096a0 tx=0x7fca6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:46.274 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.273+0000 7fca7d7fa700 1 -- 172.21.15.178:0/2693003471 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca74014420 con 0x7fca800f51a0 2023-11-15T05:03:46.481 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.481+0000 7fca85f24700 1 -- 172.21.15.178:0/2693003471 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca80107ed0 con 0x7fca800f51a0 2023-11-15T05:03:46.482 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.482+0000 7fca7d7fa700 1 -- 172.21.15.178:0/2693003471 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fca74010230 con 0x7fca800f51a0 2023-11-15T05:03:46.483 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:03:46.483 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:46.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.484+0000 7fca85f24700 1 -- 172.21.15.178:0/2693003471 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fca7002ff60 msgr2=0x7fca70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:46.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.484+0000 7fca85f24700 1 --2- 172.21.15.178:0/2693003471 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fca7002ff60 0x7fca70032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fca6c0096a0 tx=0x7fca6c006b40).stop 2023-11-15T05:03:46.484 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.484+0000 7fca85f24700 1 -- 172.21.15.178:0/2693003471 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca800f51a0 msgr2=0x7fca801064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:46.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.484+0000 7fca85f24700 1 --2- 172.21.15.178:0/2693003471 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca800f51a0 0x7fca801064a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fca74013510 tx=0x7fca7400b040).stop 2023-11-15T05:03:46.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.484+0000 7fca85f24700 1 -- 172.21.15.178:0/2693003471 shutdown_connections 2023-11-15T05:03:46.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.484+0000 7fca85f24700 1 --2- 172.21.15.178:0/2693003471 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fca7002ff60 0x7fca70032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:46.485 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.484+0000 7fca85f24700 1 --2- 172.21.15.178:0/2693003471 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca800f51a0 0x7fca801064a0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:46.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.484+0000 7fca85f24700 1 -- 172.21.15.178:0/2693003471 >> 172.21.15.178:0/2693003471 conn(0x7fca800f09a0 msgr2=0x7fca8005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:46.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.484+0000 7fca85f24700 1 -- 172.21.15.178:0/2693003471 shutdown_connections 2023-11-15T05:03:46.486 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:46.484+0000 7fca85f24700 1 -- 172.21.15.178:0/2693003471 wait complete. 2023-11-15T05:03:46.487 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:03:47.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:46 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:46.482833+0000 mon.smithi163 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.178:0/2693003471' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:47.878 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:47.878 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:48.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:47 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:46.388496+0000 mgr.smithi163.qesyrx (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:49.081 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.079+0000 7f9407108700 1 -- 172.21.15.178:0/1563337105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94000601e0 msgr2=0x7f94000f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:49.081 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.079+0000 7f9407108700 1 --2- 172.21.15.178:0/1563337105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94000601e0 0x7f94000f8f20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f93fc009fb0 tx=0x7f93fc009b30).stop 2023-11-15T05:03:49.081 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.079+0000 7f9407108700 1 -- 172.21.15.178:0/1563337105 shutdown_connections 2023-11-15T05:03:49.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.079+0000 7f9407108700 1 --2- 172.21.15.178:0/1563337105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94000601e0 0x7f94000f8f20 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:49.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.079+0000 7f9407108700 1 -- 172.21.15.178:0/1563337105 >> 172.21.15.178:0/1563337105 conn(0x7f94000f09a0 msgr2=0x7f94000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:49.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.079+0000 7f9407108700 1 -- 172.21.15.178:0/1563337105 shutdown_connections 2023-11-15T05:03:49.082 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.079+0000 7f9407108700 1 -- 172.21.15.178:0/1563337105 wait complete. 2023-11-15T05:03:49.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.079+0000 7f9407108700 1 Processor -- start 2023-11-15T05:03:49.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.080+0000 7f9407108700 1 -- start start 2023-11-15T05:03:49.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.080+0000 7f9407108700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94000601e0 0x7f9400106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:49.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.080+0000 7f9407108700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94001069f0 con 0x7f94000601e0 2023-11-15T05:03:49.083 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.080+0000 7f9406106700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94000601e0 0x7f9400106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:49.084 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.080+0000 7f9406106700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94000601e0 0x7f9400106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:32798/0 (socket says 172.21.15.178:32798) 2023-11-15T05:03:49.084 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.080+0000 7f9406106700 1 -- 172.21.15.178:0/812145882 learned_addr learned my addr 172.21.15.178:0/812145882 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:03:49.084 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.080+0000 7f9406106700 1 -- 172.21.15.178:0/812145882 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93fc005040 con 0x7f94000601e0 2023-11-15T05:03:49.084 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.080+0000 7f9406106700 1 --2- 172.21.15.178:0/812145882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94000601e0 0x7f9400106500 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f93fc013510 tx=0x7f93fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:49.084 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.081+0000 7f93f6ffd700 1 -- 172.21.15.178:0/812145882 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93fc009320 con 0x7f94000601e0 2023-11-15T05:03:49.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.081+0000 7f9407108700 1 -- 172.21.15.178:0/812145882 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9400106bf0 con 0x7f94000601e0 2023-11-15T05:03:49.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.081+0000 7f93f6ffd700 1 -- 172.21.15.178:0/812145882 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93fc013f00 con 0x7f94000601e0 2023-11-15T05:03:49.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.081+0000 7f93f6ffd700 1 -- 172.21.15.178:0/812145882 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93fc014d90 con 0x7f94000601e0 2023-11-15T05:03:49.085 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.081+0000 7f9407108700 1 -- 172.21.15.178:0/812145882 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9400107080 con 0x7f94000601e0 2023-11-15T05:03:49.086 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.081+0000 7f93f6ffd700 1 -- 172.21.15.178:0/812145882 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93fc014420 con 0x7f94000601e0 2023-11-15T05:03:49.087 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.082+0000 7f9407108700 1 -- 172.21.15.178:0/812145882 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9400045ad0 con 0x7f94000601e0 2023-11-15T05:03:49.087 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.082+0000 7f93f6ffd700 1 --2- 172.21.15.178:0/812145882 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f93ec02ff60 0x7f93ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:49.087 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.082+0000 7f93f6ffd700 1 -- 172.21.15.178:0/812145882 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f93fc041070 con 0x7f94000601e0 2023-11-15T05:03:49.087 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.082+0000 7f9405905700 1 --2- 172.21.15.178:0/812145882 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f93ec02ff60 0x7f93ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:49.087 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.082+0000 7f9405905700 1 --2- 172.21.15.178:0/812145882 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f93ec02ff60 0x7f93ec032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f93f00096a0 tx=0x7f93f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:49.088 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.087+0000 7f93f6ffd700 1 -- 172.21.15.178:0/812145882 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f93fc0095f0 con 0x7f94000601e0 2023-11-15T05:03:49.292 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.292+0000 7f9407108700 1 -- 172.21.15.178:0/812145882 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9400107f30 con 0x7f94000601e0 2023-11-15T05:03:49.293 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.293+0000 7f93f6ffd700 1 -- 172.21.15.178:0/812145882 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f93fc01db70 con 0x7f94000601e0 2023-11-15T05:03:49.293 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:03:49.294 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:49.295 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.294+0000 7f9407108700 1 -- 172.21.15.178:0/812145882 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f93ec02ff60 msgr2=0x7f93ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:49.295 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.294+0000 7f9407108700 1 --2- 172.21.15.178:0/812145882 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f93ec02ff60 0x7f93ec032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f93f00096a0 tx=0x7f93f0006b40).stop 2023-11-15T05:03:49.295 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.294+0000 7f9407108700 1 -- 172.21.15.178:0/812145882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94000601e0 msgr2=0x7f9400106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:49.295 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.294+0000 7f9407108700 1 --2- 172.21.15.178:0/812145882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94000601e0 0x7f9400106500 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f93fc013510 tx=0x7f93fc00b040).stop 2023-11-15T05:03:49.296 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.294+0000 7f9407108700 1 -- 172.21.15.178:0/812145882 shutdown_connections 2023-11-15T05:03:49.296 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.294+0000 7f9407108700 1 --2- 172.21.15.178:0/812145882 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f93ec02ff60 0x7f93ec032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:49.296 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.294+0000 7f9407108700 1 --2- 172.21.15.178:0/812145882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94000601e0 0x7f9400106500 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:49.296 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.294+0000 7f9407108700 1 -- 172.21.15.178:0/812145882 >> 172.21.15.178:0/812145882 conn(0x7f94000f09a0 msgr2=0x7f94000f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:49.296 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.294+0000 7f9407108700 1 -- 172.21.15.178:0/812145882 shutdown_connections 2023-11-15T05:03:49.297 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:49.294+0000 7f9407108700 1 -- 172.21.15.178:0/812145882 wait complete. 2023-11-15T05:03:49.297 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:03:50.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:49 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:48.388766+0000 mgr.smithi163.qesyrx (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:50.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:49 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:49.293514+0000 mon.smithi163 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.178:0/812145882' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:50.718 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:50.719 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:51.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.890+0000 7f18022f4700 1 -- 172.21.15.178:0/259927073 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17fc0f5410 msgr2=0x7f17fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:51.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.890+0000 7f18022f4700 1 --2- 172.21.15.178:0/259927073 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17fc0f5410 0x7f17fc0f5830 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f17f0009fb0 tx=0x7f17f0009b30).stop 2023-11-15T05:03:51.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.890+0000 7f18022f4700 1 -- 172.21.15.178:0/259927073 shutdown_connections 2023-11-15T05:03:51.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.890+0000 7f18022f4700 1 --2- 172.21.15.178:0/259927073 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17fc0f5410 0x7f17fc0f5830 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:51.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.890+0000 7f18022f4700 1 -- 172.21.15.178:0/259927073 >> 172.21.15.178:0/259927073 conn(0x7f17fc0f0b70 msgr2=0x7f17fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:51.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.890+0000 7f18022f4700 1 -- 172.21.15.178:0/259927073 shutdown_connections 2023-11-15T05:03:51.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.890+0000 7f18022f4700 1 -- 172.21.15.178:0/259927073 wait complete. 2023-11-15T05:03:51.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.891+0000 7f18022f4700 1 Processor -- start 2023-11-15T05:03:51.893 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.891+0000 7f18022f4700 1 -- start start 2023-11-15T05:03:51.895 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.891+0000 7f18022f4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17fc0f5410 0x7f17fc105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:51.895 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.891+0000 7f18022f4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc105af0 con 0x7f17fc0f5410 2023-11-15T05:03:51.895 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.891+0000 7f17fbfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17fc0f5410 0x7f17fc105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:51.895 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.891+0000 7f17fbfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17fc0f5410 0x7f17fc105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:32812/0 (socket says 172.21.15.178:32812) 2023-11-15T05:03:51.895 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.891+0000 7f17fbfff700 1 -- 172.21.15.178:0/446128968 learned_addr learned my addr 172.21.15.178:0/446128968 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:03:51.896 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.892+0000 7f17fbfff700 1 -- 172.21.15.178:0/446128968 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f0005040 con 0x7f17fc0f5410 2023-11-15T05:03:51.896 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.892+0000 7f17fbfff700 1 --2- 172.21.15.178:0/446128968 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17fc0f5410 0x7f17fc105600 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f17f0013510 tx=0x7f17f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:51.896 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.892+0000 7f17f8ff9700 1 -- 172.21.15.178:0/446128968 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17f0009320 con 0x7f17fc0f5410 2023-11-15T05:03:51.897 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.892+0000 7f18022f4700 1 -- 172.21.15.178:0/446128968 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17fc105cf0 con 0x7f17fc0f5410 2023-11-15T05:03:51.897 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.892+0000 7f17f8ff9700 1 -- 172.21.15.178:0/446128968 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17f0013f00 con 0x7f17fc0f5410 2023-11-15T05:03:51.897 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.892+0000 7f17f8ff9700 1 -- 172.21.15.178:0/446128968 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17f0014d90 con 0x7f17fc0f5410 2023-11-15T05:03:51.898 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.892+0000 7f18022f4700 1 -- 172.21.15.178:0/446128968 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17fc106180 con 0x7f17fc0f5410 2023-11-15T05:03:51.898 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.893+0000 7f17f8ff9700 1 -- 172.21.15.178:0/446128968 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f17f00263a0 con 0x7f17fc0f5410 2023-11-15T05:03:51.898 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.893+0000 7f17f8ff9700 1 --2- 172.21.15.178:0/446128968 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f17e402ff60 0x7f17e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:51.898 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.893+0000 7f17f8ff9700 1 -- 172.21.15.178:0/446128968 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f17f00410c0 con 0x7f17fc0f5410 2023-11-15T05:03:51.898 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.893+0000 7f18022f4700 1 -- 172.21.15.178:0/446128968 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17fc045ad0 con 0x7f17fc0f5410 2023-11-15T05:03:51.899 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.893+0000 7f17fb7fe700 1 --2- 172.21.15.178:0/446128968 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f17e402ff60 0x7f17e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:51.899 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.894+0000 7f17fb7fe700 1 --2- 172.21.15.178:0/446128968 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f17e402ff60 0x7f17e4032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f17ec0096a0 tx=0x7f17ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:51.901 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:51.898+0000 7f17f8ff9700 1 -- 172.21.15.178:0/446128968 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f17f0012920 con 0x7f17fc0f5410 2023-11-15T05:03:52.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:52.102+0000 7f18022f4700 1 -- 172.21.15.178:0/446128968 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f17fc106e60 con 0x7f17fc0f5410 2023-11-15T05:03:52.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:52.103+0000 7f17f8ff9700 1 -- 172.21.15.178:0/446128968 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f17f002b020 con 0x7f17fc0f5410 2023-11-15T05:03:52.104 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:03:52.104 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:52.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:52.104+0000 7f18022f4700 1 -- 172.21.15.178:0/446128968 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f17e402ff60 msgr2=0x7f17e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:52.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:52.104+0000 7f18022f4700 1 --2- 172.21.15.178:0/446128968 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f17e402ff60 0x7f17e4032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f17ec0096a0 tx=0x7f17ec006b40).stop 2023-11-15T05:03:52.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:52.104+0000 7f18022f4700 1 -- 172.21.15.178:0/446128968 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17fc0f5410 msgr2=0x7f17fc105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:52.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:52.104+0000 7f18022f4700 1 --2- 172.21.15.178:0/446128968 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17fc0f5410 0x7f17fc105600 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f17f0013510 tx=0x7f17f000b040).stop 2023-11-15T05:03:52.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:52.104+0000 7f18022f4700 1 -- 172.21.15.178:0/446128968 shutdown_connections 2023-11-15T05:03:52.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:52.104+0000 7f18022f4700 1 --2- 172.21.15.178:0/446128968 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f17e402ff60 0x7f17e4032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:52.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:52.104+0000 7f18022f4700 1 --2- 172.21.15.178:0/446128968 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17fc0f5410 0x7f17fc105600 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:52.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:52.104+0000 7f18022f4700 1 -- 172.21.15.178:0/446128968 >> 172.21.15.178:0/446128968 conn(0x7f17fc0f0b70 msgr2=0x7f17fc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:52.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:52.104+0000 7f18022f4700 1 -- 172.21.15.178:0/446128968 shutdown_connections 2023-11-15T05:03:52.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:52.104+0000 7f18022f4700 1 -- 172.21.15.178:0/446128968 wait complete. 2023-11-15T05:03:52.107 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:03:52.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:51 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:50.389142+0000 mgr.smithi163.qesyrx (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:53.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:52 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:52.103717+0000 mon.smithi163 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.178:0/446128968' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:53.498 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:53.498 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:54.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:53 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:52.389393+0000 mgr.smithi163.qesyrx (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:54.600 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.597+0000 7f4ab8062700 1 -- 172.21.15.178:0/1657707920 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ab00f7b60 msgr2=0x7f4ab00f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:54.601 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.597+0000 7f4ab8062700 1 --2- 172.21.15.178:0/1657707920 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ab00f7b60 0x7f4ab00f7f80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f4aac009fb0 tx=0x7f4aac009b30).stop 2023-11-15T05:03:54.601 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.598+0000 7f4ab8062700 1 -- 172.21.15.178:0/1657707920 shutdown_connections 2023-11-15T05:03:54.601 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.598+0000 7f4ab8062700 1 --2- 172.21.15.178:0/1657707920 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ab00f7b60 0x7f4ab00f7f80 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:54.601 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.598+0000 7f4ab8062700 1 -- 172.21.15.178:0/1657707920 >> 172.21.15.178:0/1657707920 conn(0x7f4ab00f02e0 msgr2=0x7f4ab00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:54.601 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.598+0000 7f4ab8062700 1 -- 172.21.15.178:0/1657707920 shutdown_connections 2023-11-15T05:03:54.602 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.598+0000 7f4ab8062700 1 -- 172.21.15.178:0/1657707920 wait complete. 2023-11-15T05:03:54.602 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.598+0000 7f4ab8062700 1 Processor -- start 2023-11-15T05:03:54.602 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.598+0000 7f4ab8062700 1 -- start start 2023-11-15T05:03:54.602 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.598+0000 7f4ab8062700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ab00f7b60 0x7f4ab00f7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:54.603 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.598+0000 7f4ab8062700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab00f58b0 con 0x7f4ab00f7b60 2023-11-15T05:03:54.603 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.598+0000 7f4ab5dfe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ab00f7b60 0x7f4ab00f7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:54.603 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.598+0000 7f4ab5dfe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ab00f7b60 0x7f4ab00f7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54978/0 (socket says 172.21.15.178:54978) 2023-11-15T05:03:54.603 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.599+0000 7f4ab5dfe700 1 -- 172.21.15.178:0/3142254846 learned_addr learned my addr 172.21.15.178:0/3142254846 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:03:54.604 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.599+0000 7f4ab5dfe700 1 -- 172.21.15.178:0/3142254846 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac005040 con 0x7f4ab00f7b60 2023-11-15T05:03:54.604 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.599+0000 7f4ab5dfe700 1 --2- 172.21.15.178:0/3142254846 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ab00f7b60 0x7f4ab00f7430 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f4aac013510 tx=0x7f4aac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:54.604 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.599+0000 7f4aa67fc700 1 -- 172.21.15.178:0/3142254846 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4aac009320 con 0x7f4ab00f7b60 2023-11-15T05:03:54.604 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.599+0000 7f4ab8062700 1 -- 172.21.15.178:0/3142254846 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab00f5a50 con 0x7f4ab00f7b60 2023-11-15T05:03:54.604 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.599+0000 7f4aa67fc700 1 -- 172.21.15.178:0/3142254846 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4aac013f00 con 0x7f4ab00f7b60 2023-11-15T05:03:54.605 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.599+0000 7f4aa67fc700 1 -- 172.21.15.178:0/3142254846 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4aac014d90 con 0x7f4ab00f7b60 2023-11-15T05:03:54.605 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.599+0000 7f4ab8062700 1 -- 172.21.15.178:0/3142254846 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab00f5ee0 con 0x7f4ab00f7b60 2023-11-15T05:03:54.606 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.600+0000 7f4aa67fc700 1 -- 172.21.15.178:0/3142254846 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4aac014420 con 0x7f4ab00f7b60 2023-11-15T05:03:54.606 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.600+0000 7f4ab8062700 1 -- 172.21.15.178:0/3142254846 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab0059100 con 0x7f4ab00f7b60 2023-11-15T05:03:54.606 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.600+0000 7f4aa67fc700 1 --2- 172.21.15.178:0/3142254846 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4a9c02ff60 0x7f4a9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:54.607 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.600+0000 7f4aa67fc700 1 -- 172.21.15.178:0/3142254846 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4aac040fb0 con 0x7f4ab00f7b60 2023-11-15T05:03:54.607 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.601+0000 7f4ab55fd700 1 --2- 172.21.15.178:0/3142254846 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4a9c02ff60 0x7f4a9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:54.607 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.601+0000 7f4ab55fd700 1 --2- 172.21.15.178:0/3142254846 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4a9c02ff60 0x7f4a9c032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f4aa00096a0 tx=0x7f4aa0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:54.608 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.605+0000 7f4aa67fc700 1 -- 172.21.15.178:0/3142254846 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4aac0095f0 con 0x7f4ab00f7b60 2023-11-15T05:03:54.816 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.814+0000 7f4ab8062700 1 -- 172.21.15.178:0/3142254846 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ab00fbf60 con 0x7f4ab00f7b60 2023-11-15T05:03:54.817 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.815+0000 7f4aa67fc700 1 -- 172.21.15.178:0/3142254846 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4aac01db70 con 0x7f4ab00f7b60 2023-11-15T05:03:54.817 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:03:54.817 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:54.818 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.816+0000 7f4ab8062700 1 -- 172.21.15.178:0/3142254846 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4a9c02ff60 msgr2=0x7f4a9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:54.819 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.816+0000 7f4ab8062700 1 --2- 172.21.15.178:0/3142254846 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4a9c02ff60 0x7f4a9c032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f4aa00096a0 tx=0x7f4aa0006b40).stop 2023-11-15T05:03:54.819 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.816+0000 7f4ab8062700 1 -- 172.21.15.178:0/3142254846 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ab00f7b60 msgr2=0x7f4ab00f7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:54.819 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.816+0000 7f4ab8062700 1 --2- 172.21.15.178:0/3142254846 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ab00f7b60 0x7f4ab00f7430 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f4aac013510 tx=0x7f4aac00b040).stop 2023-11-15T05:03:54.819 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.816+0000 7f4ab8062700 1 -- 172.21.15.178:0/3142254846 shutdown_connections 2023-11-15T05:03:54.820 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.816+0000 7f4ab8062700 1 --2- 172.21.15.178:0/3142254846 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4a9c02ff60 0x7f4a9c032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:54.820 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.816+0000 7f4ab8062700 1 --2- 172.21.15.178:0/3142254846 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ab00f7b60 0x7f4ab00f7430 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:54.820 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.816+0000 7f4ab8062700 1 -- 172.21.15.178:0/3142254846 >> 172.21.15.178:0/3142254846 conn(0x7f4ab00f02e0 msgr2=0x7f4ab00f0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:54.820 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.816+0000 7f4ab8062700 1 -- 172.21.15.178:0/3142254846 shutdown_connections 2023-11-15T05:03:54.820 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:54.816+0000 7f4ab8062700 1 -- 172.21.15.178:0/3142254846 wait complete. 2023-11-15T05:03:54.820 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:03:55.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:54 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:54.816548+0000 mon.smithi163 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.178:0/3142254846' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:56.201 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:56.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:56.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:55 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:54.389660+0000 mgr.smithi163.qesyrx (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:57.434 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.432+0000 7f7c222f2700 1 -- 172.21.15.178:0/2703379973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c1c0f62a0 msgr2=0x7f7c1c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:57.434 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.432+0000 7f7c222f2700 1 --2- 172.21.15.178:0/2703379973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c1c0f62a0 0x7f7c1c0f66c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f7c0c009fb0 tx=0x7f7c0c009b30).stop 2023-11-15T05:03:57.434 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.432+0000 7f7c222f2700 1 -- 172.21.15.178:0/2703379973 shutdown_connections 2023-11-15T05:03:57.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.432+0000 7f7c222f2700 1 --2- 172.21.15.178:0/2703379973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c1c0f62a0 0x7f7c1c0f66c0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:57.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.432+0000 7f7c222f2700 1 -- 172.21.15.178:0/2703379973 >> 172.21.15.178:0/2703379973 conn(0x7f7c1c0f1aa0 msgr2=0x7f7c1c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:57.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.432+0000 7f7c222f2700 1 -- 172.21.15.178:0/2703379973 shutdown_connections 2023-11-15T05:03:57.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.432+0000 7f7c222f2700 1 -- 172.21.15.178:0/2703379973 wait complete. 2023-11-15T05:03:57.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.432+0000 7f7c222f2700 1 Processor -- start 2023-11-15T05:03:57.436 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.432+0000 7f7c222f2700 1 -- start start 2023-11-15T05:03:57.436 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.433+0000 7f7c222f2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c1c0f62a0 0x7f7c1c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:57.436 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.433+0000 7f7c222f2700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1c1069a0 con 0x7f7c1c0f62a0 2023-11-15T05:03:57.437 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.433+0000 7f7c212f0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c1c0f62a0 0x7f7c1c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:57.438 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.433+0000 7f7c212f0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c1c0f62a0 0x7f7c1c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:54992/0 (socket says 172.21.15.178:54992) 2023-11-15T05:03:57.438 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.433+0000 7f7c212f0700 1 -- 172.21.15.178:0/911778761 learned_addr learned my addr 172.21.15.178:0/911778761 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:03:57.438 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.433+0000 7f7c212f0700 1 -- 172.21.15.178:0/911778761 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0c005040 con 0x7f7c1c0f62a0 2023-11-15T05:03:57.438 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.433+0000 7f7c212f0700 1 --2- 172.21.15.178:0/911778761 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c1c0f62a0 0x7f7c1c1064b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f7c0c013510 tx=0x7f7c0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:57.439 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.434+0000 7f7c11ffb700 1 -- 172.21.15.178:0/911778761 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c0c009320 con 0x7f7c1c0f62a0 2023-11-15T05:03:57.439 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.434+0000 7f7c222f2700 1 -- 172.21.15.178:0/911778761 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c1c106ba0 con 0x7f7c1c0f62a0 2023-11-15T05:03:57.439 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.434+0000 7f7c11ffb700 1 -- 172.21.15.178:0/911778761 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c0c013f00 con 0x7f7c1c0f62a0 2023-11-15T05:03:57.439 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.434+0000 7f7c11ffb700 1 -- 172.21.15.178:0/911778761 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c0c014dc0 con 0x7f7c1c0f62a0 2023-11-15T05:03:57.439 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.434+0000 7f7c222f2700 1 -- 172.21.15.178:0/911778761 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c1c107030 con 0x7f7c1c0f62a0 2023-11-15T05:03:57.441 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.435+0000 7f7c222f2700 1 -- 172.21.15.178:0/911778761 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c1c045ad0 con 0x7f7c1c0f62a0 2023-11-15T05:03:57.441 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.435+0000 7f7c11ffb700 1 -- 172.21.15.178:0/911778761 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7c0c0095f0 con 0x7f7c1c0f62a0 2023-11-15T05:03:57.441 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.435+0000 7f7c11ffb700 1 --2- 172.21.15.178:0/911778761 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7c0802ff60 0x7f7c08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:57.442 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.435+0000 7f7c11ffb700 1 -- 172.21.15.178:0/911778761 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7c0c041020 con 0x7f7c1c0f62a0 2023-11-15T05:03:57.442 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.435+0000 7f7c20aef700 1 --2- 172.21.15.178:0/911778761 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7c0802ff60 0x7f7c08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:57.442 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.435+0000 7f7c20aef700 1 --2- 172.21.15.178:0/911778761 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7c0802ff60 0x7f7c08032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f7c180096a0 tx=0x7f7c18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:57.443 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.440+0000 7f7c11ffb700 1 -- 172.21.15.178:0/911778761 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7c0c014420 con 0x7f7c1c0f62a0 2023-11-15T05:03:57.644 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.643+0000 7f7c222f2700 1 -- 172.21.15.178:0/911778761 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c1c107ee0 con 0x7f7c1c0f62a0 2023-11-15T05:03:57.645 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.644+0000 7f7c11ffb700 1 -- 172.21.15.178:0/911778761 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7c0c01c920 con 0x7f7c1c0f62a0 2023-11-15T05:03:57.646 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:03:57.646 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:57.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.645+0000 7f7c222f2700 1 -- 172.21.15.178:0/911778761 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7c0802ff60 msgr2=0x7f7c08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:57.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.645+0000 7f7c222f2700 1 --2- 172.21.15.178:0/911778761 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7c0802ff60 0x7f7c08032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f7c180096a0 tx=0x7f7c18006b40).stop 2023-11-15T05:03:57.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.645+0000 7f7c222f2700 1 -- 172.21.15.178:0/911778761 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c1c0f62a0 msgr2=0x7f7c1c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:57.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.645+0000 7f7c222f2700 1 --2- 172.21.15.178:0/911778761 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c1c0f62a0 0x7f7c1c1064b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f7c0c013510 tx=0x7f7c0c00b040).stop 2023-11-15T05:03:57.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.646+0000 7f7c222f2700 1 -- 172.21.15.178:0/911778761 shutdown_connections 2023-11-15T05:03:57.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.646+0000 7f7c222f2700 1 --2- 172.21.15.178:0/911778761 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f7c0802ff60 0x7f7c08032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:57.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.646+0000 7f7c222f2700 1 --2- 172.21.15.178:0/911778761 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c1c0f62a0 0x7f7c1c1064b0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:57.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.646+0000 7f7c222f2700 1 -- 172.21.15.178:0/911778761 >> 172.21.15.178:0/911778761 conn(0x7f7c1c0f1aa0 msgr2=0x7f7c1c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:57.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.646+0000 7f7c222f2700 1 -- 172.21.15.178:0/911778761 shutdown_connections 2023-11-15T05:03:57.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:03:57.646+0000 7f7c222f2700 1 -- 172.21.15.178:0/911778761 wait complete. 2023-11-15T05:03:57.649 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:03:58.205 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:57 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:56.390072+0000 mgr.smithi163.qesyrx (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:58.206 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:57 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:03:57.645687+0000 mon.smithi163 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.178:0/911778761' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:59.076 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:59.077 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:00.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:03:59 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:03:58.390471+0000 mgr.smithi163.qesyrx (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:00.216 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.213+0000 7f3f10a11700 1 -- 172.21.15.178:0/167246586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f0c0f5410 msgr2=0x7f3f0c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:00.217 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.213+0000 7f3f10a11700 1 --2- 172.21.15.178:0/167246586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f0c0f5410 0x7f3f0c0f5830 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f3efc009fb0 tx=0x7f3efc009b30).stop 2023-11-15T05:04:00.217 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.213+0000 7f3f10a11700 1 -- 172.21.15.178:0/167246586 shutdown_connections 2023-11-15T05:04:00.217 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.213+0000 7f3f10a11700 1 --2- 172.21.15.178:0/167246586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f0c0f5410 0x7f3f0c0f5830 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:00.217 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.213+0000 7f3f10a11700 1 -- 172.21.15.178:0/167246586 >> 172.21.15.178:0/167246586 conn(0x7f3f0c0f0b70 msgr2=0x7f3f0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:00.218 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.214+0000 7f3f10a11700 1 -- 172.21.15.178:0/167246586 shutdown_connections 2023-11-15T05:04:00.218 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.214+0000 7f3f10a11700 1 -- 172.21.15.178:0/167246586 wait complete. 2023-11-15T05:04:00.218 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.214+0000 7f3f10a11700 1 Processor -- start 2023-11-15T05:04:00.218 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.214+0000 7f3f10a11700 1 -- start start 2023-11-15T05:04:00.218 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.214+0000 7f3f10a11700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f0c0f5410 0x7f3f0c0fb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:00.219 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.214+0000 7f3f10a11700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c0f9a00 con 0x7f3f0c0f5410 2023-11-15T05:04:00.219 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.214+0000 7f3f0a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f0c0f5410 0x7f3f0c0fb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:00.219 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.214+0000 7f3f0a59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f0c0f5410 0x7f3f0c0fb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:55008/0 (socket says 172.21.15.178:55008) 2023-11-15T05:04:00.219 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.214+0000 7f3f0a59c700 1 -- 172.21.15.178:0/1599755093 learned_addr learned my addr 172.21.15.178:0/1599755093 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:04:00.220 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.215+0000 7f3f0a59c700 1 -- 172.21.15.178:0/1599755093 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3efc005040 con 0x7f3f0c0f5410 2023-11-15T05:04:00.220 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.215+0000 7f3f0a59c700 1 --2- 172.21.15.178:0/1599755093 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f0c0f5410 0x7f3f0c0fb580 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f3efc013510 tx=0x7f3efc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:00.220 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.215+0000 7f3f02ffd700 1 -- 172.21.15.178:0/1599755093 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3efc009320 con 0x7f3f0c0f5410 2023-11-15T05:04:00.220 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.215+0000 7f3f10a11700 1 -- 172.21.15.178:0/1599755093 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f0c0f9ba0 con 0x7f3f0c0f5410 2023-11-15T05:04:00.220 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.215+0000 7f3f02ffd700 1 -- 172.21.15.178:0/1599755093 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3efc013f00 con 0x7f3f0c0f5410 2023-11-15T05:04:00.221 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.215+0000 7f3f02ffd700 1 -- 172.21.15.178:0/1599755093 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3efc014d90 con 0x7f3f0c0f5410 2023-11-15T05:04:00.221 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.215+0000 7f3f10a11700 1 -- 172.21.15.178:0/1599755093 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f0c0fa030 con 0x7f3f0c0f5410 2023-11-15T05:04:00.222 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.216+0000 7f3f02ffd700 1 -- 172.21.15.178:0/1599755093 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3efc0095f0 con 0x7f3f0c0f5410 2023-11-15T05:04:00.222 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.216+0000 7f3f10a11700 1 -- 172.21.15.178:0/1599755093 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f0c059100 con 0x7f3f0c0f5410 2023-11-15T05:04:00.222 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.216+0000 7f3f02ffd700 1 --2- 172.21.15.178:0/1599755093 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3ef802ffb0 0x7f3ef8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:00.222 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.216+0000 7f3f02ffd700 1 -- 172.21.15.178:0/1599755093 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3efc041040 con 0x7f3f0c0f5410 2023-11-15T05:04:00.222 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.216+0000 7f3f09d9b700 1 --2- 172.21.15.178:0/1599755093 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3ef802ffb0 0x7f3ef8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:00.223 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.217+0000 7f3f09d9b700 1 --2- 172.21.15.178:0/1599755093 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3ef802ffb0 0x7f3ef8032470 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f3ef40096a0 tx=0x7f3ef4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:00.223 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.221+0000 7f3f02ffd700 1 -- 172.21.15.178:0/1599755093 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3efc014420 con 0x7f3f0c0f5410 2023-11-15T05:04:00.428 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.427+0000 7f3f10a11700 1 -- 172.21.15.178:0/1599755093 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f0c0fbf90 con 0x7f3f0c0f5410 2023-11-15T05:04:00.429 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.429+0000 7f3f02ffd700 1 -- 172.21.15.178:0/1599755093 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3efc01d7f0 con 0x7f3f0c0f5410 2023-11-15T05:04:00.430 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:04:00.430 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:00.431 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.429+0000 7f3f10a11700 1 -- 172.21.15.178:0/1599755093 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3ef802ffb0 msgr2=0x7f3ef8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:00.431 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.429+0000 7f3f10a11700 1 --2- 172.21.15.178:0/1599755093 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3ef802ffb0 0x7f3ef8032470 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f3ef40096a0 tx=0x7f3ef4006b40).stop 2023-11-15T05:04:00.431 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.430+0000 7f3f10a11700 1 -- 172.21.15.178:0/1599755093 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f0c0f5410 msgr2=0x7f3f0c0fb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:00.431 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.430+0000 7f3f10a11700 1 --2- 172.21.15.178:0/1599755093 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f0c0f5410 0x7f3f0c0fb580 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f3efc013510 tx=0x7f3efc00b040).stop 2023-11-15T05:04:00.432 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.430+0000 7f3f10a11700 1 -- 172.21.15.178:0/1599755093 shutdown_connections 2023-11-15T05:04:00.432 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.430+0000 7f3f10a11700 1 --2- 172.21.15.178:0/1599755093 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f3ef802ffb0 0x7f3ef8032470 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:00.432 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.430+0000 7f3f10a11700 1 --2- 172.21.15.178:0/1599755093 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f0c0f5410 0x7f3f0c0fb580 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:00.432 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.430+0000 7f3f10a11700 1 -- 172.21.15.178:0/1599755093 >> 172.21.15.178:0/1599755093 conn(0x7f3f0c0f0b70 msgr2=0x7f3f0c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:00.432 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.430+0000 7f3f10a11700 1 -- 172.21.15.178:0/1599755093 shutdown_connections 2023-11-15T05:04:00.433 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:00.430+0000 7f3f10a11700 1 -- 172.21.15.178:0/1599755093 wait complete. 2023-11-15T05:04:00.433 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:04:01.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:00 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:04:00.429713+0000 mon.smithi163 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.178:0/1599755093' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:01.817 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:01.818 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:02.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:01 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:04:00.390844+0000 mgr.smithi163.qesyrx (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:03.039 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.037+0000 7f6a7487f700 1 -- 172.21.15.178:0/986221905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a700f5410 msgr2=0x7f6a700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:03.039 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.037+0000 7f6a7487f700 1 --2- 172.21.15.178:0/986221905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a700f5410 0x7f6a700f5830 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f6a58009fb0 tx=0x7f6a58009b30).stop 2023-11-15T05:04:03.039 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.037+0000 7f6a7487f700 1 -- 172.21.15.178:0/986221905 shutdown_connections 2023-11-15T05:04:03.040 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.037+0000 7f6a7487f700 1 --2- 172.21.15.178:0/986221905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a700f5410 0x7f6a700f5830 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:03.040 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.037+0000 7f6a7487f700 1 -- 172.21.15.178:0/986221905 >> 172.21.15.178:0/986221905 conn(0x7f6a700f0b70 msgr2=0x7f6a700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:03.040 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.037+0000 7f6a7487f700 1 -- 172.21.15.178:0/986221905 shutdown_connections 2023-11-15T05:04:03.041 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.037+0000 7f6a7487f700 1 -- 172.21.15.178:0/986221905 wait complete. 2023-11-15T05:04:03.041 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.038+0000 7f6a7487f700 1 Processor -- start 2023-11-15T05:04:03.041 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.038+0000 7f6a7487f700 1 -- start start 2023-11-15T05:04:03.045 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.038+0000 7f6a7487f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a700f5410 0x7f6a70109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:03.046 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.038+0000 7f6a7487f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a70060240 con 0x7f6a700f5410 2023-11-15T05:04:03.046 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.038+0000 7f6a6e59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a700f5410 0x7f6a70109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:03.046 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.038+0000 7f6a6e59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a700f5410 0x7f6a70109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:55022/0 (socket says 172.21.15.178:55022) 2023-11-15T05:04:03.047 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.038+0000 7f6a6e59c700 1 -- 172.21.15.178:0/1658101774 learned_addr learned my addr 172.21.15.178:0/1658101774 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:04:03.047 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.039+0000 7f6a6e59c700 1 -- 172.21.15.178:0/1658101774 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a58005040 con 0x7f6a700f5410 2023-11-15T05:04:03.047 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.039+0000 7f6a6e59c700 1 --2- 172.21.15.178:0/1658101774 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a700f5410 0x7f6a70109430 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f6a58013510 tx=0x7f6a5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:03.047 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.039+0000 7f6a66ffd700 1 -- 172.21.15.178:0/1658101774 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a58009320 con 0x7f6a700f5410 2023-11-15T05:04:03.047 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.039+0000 7f6a66ffd700 1 -- 172.21.15.178:0/1658101774 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a58013f00 con 0x7f6a700f5410 2023-11-15T05:04:03.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.039+0000 7f6a7487f700 1 -- 172.21.15.178:0/1658101774 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a70060440 con 0x7f6a700f5410 2023-11-15T05:04:03.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.039+0000 7f6a66ffd700 1 -- 172.21.15.178:0/1658101774 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a58014d90 con 0x7f6a700f5410 2023-11-15T05:04:03.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.039+0000 7f6a7487f700 1 -- 172.21.15.178:0/1658101774 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a70109ae0 con 0x7f6a700f5410 2023-11-15T05:04:03.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.040+0000 7f6a66ffd700 1 -- 172.21.15.178:0/1658101774 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6a580095f0 con 0x7f6a700f5410 2023-11-15T05:04:03.048 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.040+0000 7f6a7487f700 1 -- 172.21.15.178:0/1658101774 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a70045ad0 con 0x7f6a700f5410 2023-11-15T05:04:03.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.040+0000 7f6a66ffd700 1 --2- 172.21.15.178:0/1658101774 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6a5c02ff60 0x7f6a5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:03.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.040+0000 7f6a66ffd700 1 -- 172.21.15.178:0/1658101774 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6a58041000 con 0x7f6a700f5410 2023-11-15T05:04:03.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.040+0000 7f6a6dd9b700 1 --2- 172.21.15.178:0/1658101774 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6a5c02ff60 0x7f6a5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:03.049 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.041+0000 7f6a6dd9b700 1 --2- 172.21.15.178:0/1658101774 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6a5c02ff60 0x7f6a5c032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f6a600096a0 tx=0x7f6a60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:03.050 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.045+0000 7f6a66ffd700 1 -- 172.21.15.178:0/1658101774 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a5801d7f0 con 0x7f6a700f5410 2023-11-15T05:04:03.251 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.250+0000 7f6a7487f700 1 -- 172.21.15.178:0/1658101774 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6a700fae60 con 0x7f6a700f5410 2023-11-15T05:04:03.252 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.250+0000 7f6a66ffd700 1 -- 172.21.15.178:0/1658101774 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6a58010230 con 0x7f6a700f5410 2023-11-15T05:04:03.252 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:04:03.253 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:03.254 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.252+0000 7f6a7487f700 1 -- 172.21.15.178:0/1658101774 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6a5c02ff60 msgr2=0x7f6a5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:03.254 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.252+0000 7f6a7487f700 1 --2- 172.21.15.178:0/1658101774 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6a5c02ff60 0x7f6a5c032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f6a600096a0 tx=0x7f6a60006b40).stop 2023-11-15T05:04:03.254 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.252+0000 7f6a7487f700 1 -- 172.21.15.178:0/1658101774 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a700f5410 msgr2=0x7f6a70109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:03.254 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.252+0000 7f6a7487f700 1 --2- 172.21.15.178:0/1658101774 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a700f5410 0x7f6a70109430 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f6a58013510 tx=0x7f6a5800b040).stop 2023-11-15T05:04:03.255 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.252+0000 7f6a7487f700 1 -- 172.21.15.178:0/1658101774 shutdown_connections 2023-11-15T05:04:03.255 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.252+0000 7f6a7487f700 1 --2- 172.21.15.178:0/1658101774 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f6a5c02ff60 0x7f6a5c032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:03.255 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.252+0000 7f6a7487f700 1 --2- 172.21.15.178:0/1658101774 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a700f5410 0x7f6a70109430 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:03.255 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.252+0000 7f6a7487f700 1 -- 172.21.15.178:0/1658101774 >> 172.21.15.178:0/1658101774 conn(0x7f6a700f0b70 msgr2=0x7f6a700f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:03.255 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.252+0000 7f6a7487f700 1 -- 172.21.15.178:0/1658101774 shutdown_connections 2023-11-15T05:04:03.256 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:03.252+0000 7f6a7487f700 1 -- 172.21.15.178:0/1658101774 wait complete. 2023-11-15T05:04:03.256 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:04:04.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:03 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:04:02.391209+0000 mgr.smithi163.qesyrx (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:04.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:03 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:04:03.252250+0000 mon.smithi163 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.178:0/1658101774' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:04.657 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:04.658 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:05.887 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.885+0000 7f44ae2d5700 1 -- 172.21.15.178:0/2044025671 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44a8060180 msgr2=0x7f44a80f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:05.888 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.885+0000 7f44ae2d5700 1 --2- 172.21.15.178:0/2044025671 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44a8060180 0x7f44a80f8f80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f44a4009fb0 tx=0x7f44a4009b30).stop 2023-11-15T05:04:05.888 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.885+0000 7f44ae2d5700 1 -- 172.21.15.178:0/2044025671 shutdown_connections 2023-11-15T05:04:05.888 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.885+0000 7f44ae2d5700 1 --2- 172.21.15.178:0/2044025671 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44a8060180 0x7f44a80f8f80 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:05.888 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.885+0000 7f44ae2d5700 1 -- 172.21.15.178:0/2044025671 >> 172.21.15.178:0/2044025671 conn(0x7f44a80f09a0 msgr2=0x7f44a80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:05.889 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.885+0000 7f44ae2d5700 1 -- 172.21.15.178:0/2044025671 shutdown_connections 2023-11-15T05:04:05.889 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.885+0000 7f44ae2d5700 1 -- 172.21.15.178:0/2044025671 wait complete. 2023-11-15T05:04:05.889 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.885+0000 7f44ae2d5700 1 Processor -- start 2023-11-15T05:04:05.889 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.885+0000 7f44ae2d5700 1 -- start start 2023-11-15T05:04:05.889 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.885+0000 7f44ae2d5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44a8060180 0x7f44a8106530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:05.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.885+0000 7f44ae2d5700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a8106a20 con 0x7f44a8060180 2023-11-15T05:04:05.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.886+0000 7f44ad2d3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44a8060180 0x7f44a8106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:05.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.886+0000 7f44ad2d3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44a8060180 0x7f44a8106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:58240/0 (socket says 172.21.15.178:58240) 2023-11-15T05:04:05.890 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.886+0000 7f44ad2d3700 1 -- 172.21.15.178:0/284325707 learned_addr learned my addr 172.21.15.178:0/284325707 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:04:05.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.886+0000 7f44ad2d3700 1 -- 172.21.15.178:0/284325707 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a4005040 con 0x7f44a8060180 2023-11-15T05:04:05.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.886+0000 7f44ad2d3700 1 --2- 172.21.15.178:0/284325707 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44a8060180 0x7f44a8106530 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f44a4013510 tx=0x7f44a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:05.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.886+0000 7f449dffb700 1 -- 172.21.15.178:0/284325707 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44a4009320 con 0x7f44a8060180 2023-11-15T05:04:05.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.886+0000 7f44ae2d5700 1 -- 172.21.15.178:0/284325707 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a8106c20 con 0x7f44a8060180 2023-11-15T05:04:05.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.886+0000 7f449dffb700 1 -- 172.21.15.178:0/284325707 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44a4013f00 con 0x7f44a8060180 2023-11-15T05:04:05.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.886+0000 7f449dffb700 1 -- 172.21.15.178:0/284325707 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44a4014d90 con 0x7f44a8060180 2023-11-15T05:04:05.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.886+0000 7f44ae2d5700 1 -- 172.21.15.178:0/284325707 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a81070b0 con 0x7f44a8060180 2023-11-15T05:04:05.893 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.887+0000 7f449dffb700 1 -- 172.21.15.178:0/284325707 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44a40263a0 con 0x7f44a8060180 2023-11-15T05:04:05.893 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.887+0000 7f44ae2d5700 1 -- 172.21.15.178:0/284325707 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a8045ad0 con 0x7f44a8060180 2023-11-15T05:04:05.894 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.887+0000 7f449dffb700 1 --2- 172.21.15.178:0/284325707 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f449402ff60 0x7f4494032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:05.895 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.887+0000 7f449dffb700 1 -- 172.21.15.178:0/284325707 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f44a40412c0 con 0x7f44a8060180 2023-11-15T05:04:05.895 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.888+0000 7f44acad2700 1 --2- 172.21.15.178:0/284325707 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f449402ff60 0x7f4494032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:05.895 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.888+0000 7f44acad2700 1 --2- 172.21.15.178:0/284325707 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f449402ff60 0x7f4494032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f44980096a0 tx=0x7f4498006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:05.895 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:05.892+0000 7f449dffb700 1 -- 172.21.15.178:0/284325707 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44a40126f0 con 0x7f44a8060180 2023-11-15T05:04:06.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:06.096+0000 7f44ae2d5700 1 -- 172.21.15.178:0/284325707 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44a8107f60 con 0x7f44a8060180 2023-11-15T05:04:06.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:06.097+0000 7f449dffb700 1 -- 172.21.15.178:0/284325707 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f44a4012330 con 0x7f44a8060180 2023-11-15T05:04:06.099 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:04:06.099 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:06.100 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:06.099+0000 7f44ae2d5700 1 -- 172.21.15.178:0/284325707 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f449402ff60 msgr2=0x7f4494032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:06.100 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:06.099+0000 7f44ae2d5700 1 --2- 172.21.15.178:0/284325707 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f449402ff60 0x7f4494032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f44980096a0 tx=0x7f4498006b40).stop 2023-11-15T05:04:06.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:06.099+0000 7f44ae2d5700 1 -- 172.21.15.178:0/284325707 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44a8060180 msgr2=0x7f44a8106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:06.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:06.099+0000 7f44ae2d5700 1 --2- 172.21.15.178:0/284325707 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44a8060180 0x7f44a8106530 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f44a4013510 tx=0x7f44a400b040).stop 2023-11-15T05:04:06.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:06.099+0000 7f44ae2d5700 1 -- 172.21.15.178:0/284325707 shutdown_connections 2023-11-15T05:04:06.101 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:06.099+0000 7f44ae2d5700 1 --2- 172.21.15.178:0/284325707 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f449402ff60 0x7f4494032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:06.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:06.099+0000 7f44ae2d5700 1 --2- 172.21.15.178:0/284325707 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44a8060180 0x7f44a8106530 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:06.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:06.099+0000 7f44ae2d5700 1 -- 172.21.15.178:0/284325707 >> 172.21.15.178:0/284325707 conn(0x7f44a80f09a0 msgr2=0x7f44a80f1490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:06.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:06.099+0000 7f44ae2d5700 1 -- 172.21.15.178:0/284325707 shutdown_connections 2023-11-15T05:04:06.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:06.099+0000 7f44ae2d5700 1 -- 172.21.15.178:0/284325707 wait complete. 2023-11-15T05:04:06.103 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:04:06.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:05 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:04:04.391654+0000 mgr.smithi163.qesyrx (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:07.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:06 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:04:06.098856+0000 mon.smithi163 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.178:0/284325707' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:07.474 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:07.475 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:08.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:07 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:04:06.392051+0000 mgr.smithi163.qesyrx (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:08.676 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.674+0000 7f4a34817700 1 -- 172.21.15.178:0/3133492113 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a300f5410 msgr2=0x7f4a300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:08.676 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.674+0000 7f4a34817700 1 --2- 172.21.15.178:0/3133492113 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a300f5410 0x7f4a300f5830 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f4a18009fb0 tx=0x7f4a18009b30).stop 2023-11-15T05:04:08.677 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.674+0000 7f4a34817700 1 -- 172.21.15.178:0/3133492113 shutdown_connections 2023-11-15T05:04:08.677 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.674+0000 7f4a34817700 1 --2- 172.21.15.178:0/3133492113 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a300f5410 0x7f4a300f5830 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:08.677 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.674+0000 7f4a34817700 1 -- 172.21.15.178:0/3133492113 >> 172.21.15.178:0/3133492113 conn(0x7f4a300f0b70 msgr2=0x7f4a300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:08.677 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.674+0000 7f4a34817700 1 -- 172.21.15.178:0/3133492113 shutdown_connections 2023-11-15T05:04:08.677 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.674+0000 7f4a34817700 1 -- 172.21.15.178:0/3133492113 wait complete. 2023-11-15T05:04:08.678 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.675+0000 7f4a34817700 1 Processor -- start 2023-11-15T05:04:08.678 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.675+0000 7f4a34817700 1 -- start start 2023-11-15T05:04:08.678 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.675+0000 7f4a34817700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a300f5410 0x7f4a30109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:08.678 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.675+0000 7f4a34817700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a30060220 con 0x7f4a300f5410 2023-11-15T05:04:08.680 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.675+0000 7f4a2e59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a300f5410 0x7f4a30109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:08.680 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.675+0000 7f4a2e59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a300f5410 0x7f4a30109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:58256/0 (socket says 172.21.15.178:58256) 2023-11-15T05:04:08.680 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.675+0000 7f4a2e59c700 1 -- 172.21.15.178:0/1179559761 learned_addr learned my addr 172.21.15.178:0/1179559761 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:04:08.680 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.675+0000 7f4a2e59c700 1 -- 172.21.15.178:0/1179559761 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a18005040 con 0x7f4a300f5410 2023-11-15T05:04:08.681 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.676+0000 7f4a2e59c700 1 --2- 172.21.15.178:0/1179559761 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a300f5410 0x7f4a30109420 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f4a18013510 tx=0x7f4a1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:08.681 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.676+0000 7f4a26ffd700 1 -- 172.21.15.178:0/1179559761 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a18009320 con 0x7f4a300f5410 2023-11-15T05:04:08.681 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.676+0000 7f4a34817700 1 -- 172.21.15.178:0/1179559761 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a30060420 con 0x7f4a300f5410 2023-11-15T05:04:08.681 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.676+0000 7f4a26ffd700 1 -- 172.21.15.178:0/1179559761 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a18013f00 con 0x7f4a300f5410 2023-11-15T05:04:08.681 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.676+0000 7f4a26ffd700 1 -- 172.21.15.178:0/1179559761 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a18014d90 con 0x7f4a300f5410 2023-11-15T05:04:08.681 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.676+0000 7f4a34817700 1 -- 172.21.15.178:0/1179559761 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a30109ad0 con 0x7f4a300f5410 2023-11-15T05:04:08.682 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.677+0000 7f4a26ffd700 1 -- 172.21.15.178:0/1179559761 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a180263a0 con 0x7f4a300f5410 2023-11-15T05:04:08.682 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.677+0000 7f4a26ffd700 1 --2- 172.21.15.178:0/1179559761 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4a1c02ffb0 0x7f4a1c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:08.682 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.677+0000 7f4a26ffd700 1 -- 172.21.15.178:0/1179559761 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a18041270 con 0x7f4a300f5410 2023-11-15T05:04:08.682 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.677+0000 7f4a34817700 1 -- 172.21.15.178:0/1179559761 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a30059100 con 0x7f4a300f5410 2023-11-15T05:04:08.682 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.677+0000 7f4a2dd9b700 1 --2- 172.21.15.178:0/1179559761 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4a1c02ffb0 0x7f4a1c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:08.683 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.678+0000 7f4a2dd9b700 1 --2- 172.21.15.178:0/1179559761 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4a1c02ffb0 0x7f4a1c032470 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f4a200096a0 tx=0x7f4a20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:08.683 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.682+0000 7f4a26ffd700 1 -- 172.21.15.178:0/1179559761 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a1801cb10 con 0x7f4a300f5410 2023-11-15T05:04:08.888 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.887+0000 7f4a34817700 1 -- 172.21.15.178:0/1179559761 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a3010a180 con 0x7f4a300f5410 2023-11-15T05:04:08.889 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.888+0000 7f4a26ffd700 1 -- 172.21.15.178:0/1179559761 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4a1802b030 con 0x7f4a300f5410 2023-11-15T05:04:08.889 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:04:08.890 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:08.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.889+0000 7f4a34817700 1 -- 172.21.15.178:0/1179559761 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4a1c02ffb0 msgr2=0x7f4a1c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:08.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.889+0000 7f4a34817700 1 --2- 172.21.15.178:0/1179559761 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4a1c02ffb0 0x7f4a1c032470 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f4a200096a0 tx=0x7f4a20006b40).stop 2023-11-15T05:04:08.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.889+0000 7f4a34817700 1 -- 172.21.15.178:0/1179559761 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a300f5410 msgr2=0x7f4a30109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:08.891 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.889+0000 7f4a34817700 1 --2- 172.21.15.178:0/1179559761 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a300f5410 0x7f4a30109420 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f4a18013510 tx=0x7f4a1800b040).stop 2023-11-15T05:04:08.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.889+0000 7f4a34817700 1 -- 172.21.15.178:0/1179559761 shutdown_connections 2023-11-15T05:04:08.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.889+0000 7f4a34817700 1 --2- 172.21.15.178:0/1179559761 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f4a1c02ffb0 0x7f4a1c032470 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:08.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.889+0000 7f4a34817700 1 --2- 172.21.15.178:0/1179559761 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a300f5410 0x7f4a30109420 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:08.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.889+0000 7f4a34817700 1 -- 172.21.15.178:0/1179559761 >> 172.21.15.178:0/1179559761 conn(0x7f4a300f0b70 msgr2=0x7f4a300f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:08.892 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.889+0000 7f4a34817700 1 -- 172.21.15.178:0/1179559761 shutdown_connections 2023-11-15T05:04:08.893 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:08.889+0000 7f4a34817700 1 -- 172.21.15.178:0/1179559761 wait complete. 2023-11-15T05:04:08.894 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:04:10.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:09 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:04:08.392448+0000 mgr.smithi163.qesyrx (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:10.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:09 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:04:08.889413+0000 mon.smithi163 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.178:0/1179559761' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:10.312 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:10.312 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:11.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.497+0000 7f0a09a82700 1 -- 172.21.15.178:0/1769393049 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0a040f5410 msgr2=0x7f0a040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:11.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.497+0000 7f0a09a82700 1 --2- 172.21.15.178:0/1769393049 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0a040f5410 0x7f0a040f5830 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f09f4009fb0 tx=0x7f09f4009b30).stop 2023-11-15T05:04:11.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.497+0000 7f0a09a82700 1 -- 172.21.15.178:0/1769393049 shutdown_connections 2023-11-15T05:04:11.500 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.497+0000 7f0a09a82700 1 --2- 172.21.15.178:0/1769393049 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0a040f5410 0x7f0a040f5830 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:11.500 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.497+0000 7f0a09a82700 1 -- 172.21.15.178:0/1769393049 >> 172.21.15.178:0/1769393049 conn(0x7f0a040f0b70 msgr2=0x7f0a040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:11.500 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.497+0000 7f0a09a82700 1 -- 172.21.15.178:0/1769393049 shutdown_connections 2023-11-15T05:04:11.500 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.497+0000 7f0a09a82700 1 -- 172.21.15.178:0/1769393049 wait complete. 2023-11-15T05:04:11.500 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.498+0000 7f0a09a82700 1 Processor -- start 2023-11-15T05:04:11.502 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.498+0000 7f0a09a82700 1 -- start start 2023-11-15T05:04:11.503 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.498+0000 7f0a09a82700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0a040f5410 0x7f0a04109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:11.503 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.498+0000 7f0a09a82700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a04060240 con 0x7f0a040f5410 2023-11-15T05:04:11.503 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.498+0000 7f0a037fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0a040f5410 0x7f0a04109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:11.503 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.498+0000 7f0a037fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0a040f5410 0x7f0a04109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:58268/0 (socket says 172.21.15.178:58268) 2023-11-15T05:04:11.503 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.499+0000 7f0a037fe700 1 -- 172.21.15.178:0/2793495487 learned_addr learned my addr 172.21.15.178:0/2793495487 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:04:11.504 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.499+0000 7f0a037fe700 1 -- 172.21.15.178:0/2793495487 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f4005040 con 0x7f0a040f5410 2023-11-15T05:04:11.504 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.499+0000 7f0a037fe700 1 --2- 172.21.15.178:0/2793495487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0a040f5410 0x7f0a04109430 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f09f4013510 tx=0x7f09f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:11.504 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.499+0000 7f09ebfff700 1 -- 172.21.15.178:0/2793495487 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09f4009320 con 0x7f0a040f5410 2023-11-15T05:04:11.504 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.499+0000 7f0a09a82700 1 -- 172.21.15.178:0/2793495487 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a04060440 con 0x7f0a040f5410 2023-11-15T05:04:11.504 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.499+0000 7f09ebfff700 1 -- 172.21.15.178:0/2793495487 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09f4013f00 con 0x7f0a040f5410 2023-11-15T05:04:11.505 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.499+0000 7f09ebfff700 1 -- 172.21.15.178:0/2793495487 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09f4014d90 con 0x7f0a040f5410 2023-11-15T05:04:11.505 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.499+0000 7f0a09a82700 1 -- 172.21.15.178:0/2793495487 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a04109ae0 con 0x7f0a040f5410 2023-11-15T05:04:11.505 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.500+0000 7f0a09a82700 1 -- 172.21.15.178:0/2793495487 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a04045ad0 con 0x7f0a040f5410 2023-11-15T05:04:11.505 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.500+0000 7f09ebfff700 1 -- 172.21.15.178:0/2793495487 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09f40095f0 con 0x7f0a040f5410 2023-11-15T05:04:11.505 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.500+0000 7f09ebfff700 1 --2- 172.21.15.178:0/2793495487 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f09ec02ff60 0x7f09ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:11.505 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.500+0000 7f09ebfff700 1 -- 172.21.15.178:0/2793495487 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f09f4040fc0 con 0x7f0a040f5410 2023-11-15T05:04:11.506 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.501+0000 7f0a02ffd700 1 --2- 172.21.15.178:0/2793495487 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f09ec02ff60 0x7f09ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:11.506 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.501+0000 7f0a02ffd700 1 --2- 172.21.15.178:0/2793495487 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f09ec02ff60 0x7f09ec032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f09f80096a0 tx=0x7f09f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:11.506 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.505+0000 7f09ebfff700 1 -- 172.21.15.178:0/2793495487 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09f4014420 con 0x7f0a040f5410 2023-11-15T05:04:11.710 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.709+0000 7f0a09a82700 1 -- 172.21.15.178:0/2793495487 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0a040fae60 con 0x7f0a040f5410 2023-11-15T05:04:11.711 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.710+0000 7f09ebfff700 1 -- 172.21.15.178:0/2793495487 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f09f4010230 con 0x7f0a040f5410 2023-11-15T05:04:11.711 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:04:11.711 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:11.713 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.711+0000 7f0a09a82700 1 -- 172.21.15.178:0/2793495487 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f09ec02ff60 msgr2=0x7f09ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:11.713 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.711+0000 7f0a09a82700 1 --2- 172.21.15.178:0/2793495487 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f09ec02ff60 0x7f09ec032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f09f80096a0 tx=0x7f09f8006b40).stop 2023-11-15T05:04:11.714 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.711+0000 7f0a09a82700 1 -- 172.21.15.178:0/2793495487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0a040f5410 msgr2=0x7f0a04109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:11.714 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.711+0000 7f0a09a82700 1 --2- 172.21.15.178:0/2793495487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0a040f5410 0x7f0a04109430 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f09f4013510 tx=0x7f09f400b040).stop 2023-11-15T05:04:11.714 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.711+0000 7f0a09a82700 1 -- 172.21.15.178:0/2793495487 shutdown_connections 2023-11-15T05:04:11.714 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.711+0000 7f0a09a82700 1 --2- 172.21.15.178:0/2793495487 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f09ec02ff60 0x7f09ec032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:11.714 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.711+0000 7f0a09a82700 1 --2- 172.21.15.178:0/2793495487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0a040f5410 0x7f0a04109430 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:11.715 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.711+0000 7f0a09a82700 1 -- 172.21.15.178:0/2793495487 >> 172.21.15.178:0/2793495487 conn(0x7f0a040f0b70 msgr2=0x7f0a040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:11.715 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.711+0000 7f0a09a82700 1 -- 172.21.15.178:0/2793495487 shutdown_connections 2023-11-15T05:04:11.715 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:11.711+0000 7f0a09a82700 1 -- 172.21.15.178:0/2793495487 wait complete. 2023-11-15T05:04:11.715 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:04:12.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:11 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:04:10.392854+0000 mgr.smithi163.qesyrx (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:12.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:11 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:04:11.711370+0000 mon.smithi163 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.178:0/2793495487' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:13.101 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:13.102 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:14.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:13 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:04:12.393173+0000 mgr.smithi163.qesyrx (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:14.281 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.279+0000 7f981cdf4700 1 -- 172.21.15.178:0/3272347433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9818060200 msgr2=0x7f98180f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:14.281 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.279+0000 7f981cdf4700 1 --2- 172.21.15.178:0/3272347433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9818060200 0x7f98180f9170 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f9800009fb0 tx=0x7f9800009b30).stop 2023-11-15T05:04:14.282 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.279+0000 7f981cdf4700 1 -- 172.21.15.178:0/3272347433 shutdown_connections 2023-11-15T05:04:14.282 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.279+0000 7f981cdf4700 1 --2- 172.21.15.178:0/3272347433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9818060200 0x7f98180f9170 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:14.282 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.279+0000 7f981cdf4700 1 -- 172.21.15.178:0/3272347433 >> 172.21.15.178:0/3272347433 conn(0x7f98180f0b70 msgr2=0x7f98180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:14.282 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.279+0000 7f981cdf4700 1 -- 172.21.15.178:0/3272347433 shutdown_connections 2023-11-15T05:04:14.282 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.279+0000 7f981cdf4700 1 -- 172.21.15.178:0/3272347433 wait complete. 2023-11-15T05:04:14.283 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.280+0000 7f981cdf4700 1 Processor -- start 2023-11-15T05:04:14.283 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.280+0000 7f981cdf4700 1 -- start start 2023-11-15T05:04:14.283 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.280+0000 7f981cdf4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9818060200 0x7f98180f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:14.283 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.280+0000 7f981cdf4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98180f5070 con 0x7f9818060200 2023-11-15T05:04:14.285 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.280+0000 7f981659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9818060200 0x7f98180f6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:14.285 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.280+0000 7f981659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9818060200 0x7f98180f6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:58284/0 (socket says 172.21.15.178:58284) 2023-11-15T05:04:14.285 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.280+0000 7f981659c700 1 -- 172.21.15.178:0/4106897067 learned_addr learned my addr 172.21.15.178:0/4106897067 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:04:14.285 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.280+0000 7f981659c700 1 -- 172.21.15.178:0/4106897067 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9800005040 con 0x7f9818060200 2023-11-15T05:04:14.286 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.281+0000 7f981659c700 1 --2- 172.21.15.178:0/4106897067 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9818060200 0x7f98180f6bf0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f9800013510 tx=0x7f980000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:14.286 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.281+0000 7f980effd700 1 -- 172.21.15.178:0/4106897067 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9800009320 con 0x7f9818060200 2023-11-15T05:04:14.286 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.281+0000 7f981cdf4700 1 -- 172.21.15.178:0/4106897067 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98180f5210 con 0x7f9818060200 2023-11-15T05:04:14.286 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.281+0000 7f980effd700 1 -- 172.21.15.178:0/4106897067 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9800013f00 con 0x7f9818060200 2023-11-15T05:04:14.286 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.281+0000 7f980effd700 1 -- 172.21.15.178:0/4106897067 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9800014dc0 con 0x7f9818060200 2023-11-15T05:04:14.286 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.281+0000 7f981cdf4700 1 -- 172.21.15.178:0/4106897067 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98180f56a0 con 0x7f9818060200 2023-11-15T05:04:14.287 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.282+0000 7f980effd700 1 -- 172.21.15.178:0/4106897067 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f980001d3d0 con 0x7f9818060200 2023-11-15T05:04:14.287 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.282+0000 7f980effd700 1 --2- 172.21.15.178:0/4106897067 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f980402ff60 0x7f9804032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:14.287 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.282+0000 7f981cdf4700 1 -- 172.21.15.178:0/4106897067 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9818059100 con 0x7f9818060200 2023-11-15T05:04:14.287 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.282+0000 7f980effd700 1 -- 172.21.15.178:0/4106897067 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f98000413b0 con 0x7f9818060200 2023-11-15T05:04:14.287 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.282+0000 7f9815d9b700 1 --2- 172.21.15.178:0/4106897067 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f980402ff60 0x7f9804032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:14.288 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.282+0000 7f9815d9b700 1 --2- 172.21.15.178:0/4106897067 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f980402ff60 0x7f9804032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f98080096a0 tx=0x7f9808006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:14.288 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.287+0000 7f980effd700 1 -- 172.21.15.178:0/4106897067 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f98000126f0 con 0x7f9818060200 2023-11-15T05:04:14.498 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.497+0000 7f981cdf4700 1 -- 172.21.15.178:0/4106897067 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9818045ad0 con 0x7f9818060200 2023-11-15T05:04:14.499 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.498+0000 7f980effd700 1 -- 172.21.15.178:0/4106897067 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9800012330 con 0x7f9818060200 2023-11-15T05:04:14.499 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:04:14.500 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:14.500 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.499+0000 7f981cdf4700 1 -- 172.21.15.178:0/4106897067 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f980402ff60 msgr2=0x7f9804032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:14.501 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.499+0000 7f981cdf4700 1 --2- 172.21.15.178:0/4106897067 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f980402ff60 0x7f9804032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f98080096a0 tx=0x7f9808006b40).stop 2023-11-15T05:04:14.501 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.499+0000 7f981cdf4700 1 -- 172.21.15.178:0/4106897067 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9818060200 msgr2=0x7f98180f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:14.501 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.499+0000 7f981cdf4700 1 --2- 172.21.15.178:0/4106897067 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9818060200 0x7f98180f6bf0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f9800013510 tx=0x7f980000b040).stop 2023-11-15T05:04:14.501 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.499+0000 7f981cdf4700 1 -- 172.21.15.178:0/4106897067 shutdown_connections 2023-11-15T05:04:14.501 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.499+0000 7f981cdf4700 1 --2- 172.21.15.178:0/4106897067 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f980402ff60 0x7f9804032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:14.502 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.499+0000 7f981cdf4700 1 --2- 172.21.15.178:0/4106897067 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9818060200 0x7f98180f6bf0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:14.502 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.499+0000 7f981cdf4700 1 -- 172.21.15.178:0/4106897067 >> 172.21.15.178:0/4106897067 conn(0x7f98180f0b70 msgr2=0x7f98180f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:14.502 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.499+0000 7f981cdf4700 1 -- 172.21.15.178:0/4106897067 shutdown_connections 2023-11-15T05:04:14.502 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:14.499+0000 7f981cdf4700 1 -- 172.21.15.178:0/4106897067 wait complete. 2023-11-15T05:04:14.504 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:04:15.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:14 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:04:14.499340+0000 mon.smithi163 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.178:0/4106897067' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:15.884 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:15.885 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:16.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:15 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:04:14.393630+0000 mgr.smithi163.qesyrx (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:17.059 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.057+0000 7f8c646f8700 1 -- 172.21.15.178:0/2557399585 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c5c0f5410 msgr2=0x7f8c5c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:17.059 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.057+0000 7f8c646f8700 1 --2- 172.21.15.178:0/2557399585 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c5c0f5410 0x7f8c5c0f5830 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f8c58009fb0 tx=0x7f8c58009b30).stop 2023-11-15T05:04:17.059 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.057+0000 7f8c646f8700 1 -- 172.21.15.178:0/2557399585 shutdown_connections 2023-11-15T05:04:17.059 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.057+0000 7f8c646f8700 1 --2- 172.21.15.178:0/2557399585 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c5c0f5410 0x7f8c5c0f5830 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:17.060 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.057+0000 7f8c646f8700 1 -- 172.21.15.178:0/2557399585 >> 172.21.15.178:0/2557399585 conn(0x7f8c5c0f0b70 msgr2=0x7f8c5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:17.060 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.057+0000 7f8c646f8700 1 -- 172.21.15.178:0/2557399585 shutdown_connections 2023-11-15T05:04:17.060 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.057+0000 7f8c646f8700 1 -- 172.21.15.178:0/2557399585 wait complete. 2023-11-15T05:04:17.060 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.057+0000 7f8c646f8700 1 Processor -- start 2023-11-15T05:04:17.060 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.057+0000 7f8c646f8700 1 -- start start 2023-11-15T05:04:17.061 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.057+0000 7f8c646f8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c5c0f5410 0x7f8c5c109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:17.061 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.057+0000 7f8c646f8700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5c060240 con 0x7f8c5c0f5410 2023-11-15T05:04:17.061 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.057+0000 7f8c62494700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c5c0f5410 0x7f8c5c109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:17.061 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.057+0000 7f8c62494700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c5c0f5410 0x7f8c5c109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:47816/0 (socket says 172.21.15.178:47816) 2023-11-15T05:04:17.061 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.057+0000 7f8c62494700 1 -- 172.21.15.178:0/3971375963 learned_addr learned my addr 172.21.15.178:0/3971375963 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:04:17.062 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.058+0000 7f8c62494700 1 -- 172.21.15.178:0/3971375963 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c58005040 con 0x7f8c5c0f5410 2023-11-15T05:04:17.062 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.058+0000 7f8c62494700 1 --2- 172.21.15.178:0/3971375963 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c5c0f5410 0x7f8c5c109400 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f8c58014000 tx=0x7f8c5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:17.062 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.058+0000 7f8c52ffd700 1 -- 172.21.15.178:0/3971375963 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c580095f0 con 0x7f8c5c0f5410 2023-11-15T05:04:17.062 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.058+0000 7f8c52ffd700 1 -- 172.21.15.178:0/3971375963 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c58007e70 con 0x7f8c5c0f5410 2023-11-15T05:04:17.062 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.058+0000 7f8c646f8700 1 -- 172.21.15.178:0/3971375963 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c5c060440 con 0x7f8c5c0f5410 2023-11-15T05:04:17.063 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.058+0000 7f8c52ffd700 1 -- 172.21.15.178:0/3971375963 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c58014ea0 con 0x7f8c5c0f5410 2023-11-15T05:04:17.063 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.058+0000 7f8c646f8700 1 -- 172.21.15.178:0/3971375963 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c5c109ab0 con 0x7f8c5c0f5410 2023-11-15T05:04:17.064 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.059+0000 7f8c52ffd700 1 -- 172.21.15.178:0/3971375963 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8c58025070 con 0x7f8c5c0f5410 2023-11-15T05:04:17.064 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.059+0000 7f8c646f8700 1 -- 172.21.15.178:0/3971375963 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c5c045ad0 con 0x7f8c5c0f5410 2023-11-15T05:04:17.064 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.059+0000 7f8c52ffd700 1 --2- 172.21.15.178:0/3971375963 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8c4802ffb0 0x7f8c48032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:17.065 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.059+0000 7f8c52ffd700 1 -- 172.21.15.178:0/3971375963 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8c58040920 con 0x7f8c5c0f5410 2023-11-15T05:04:17.065 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.059+0000 7f8c61c93700 1 --2- 172.21.15.178:0/3971375963 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8c4802ffb0 0x7f8c48032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:17.065 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.060+0000 7f8c61c93700 1 --2- 172.21.15.178:0/3971375963 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8c4802ffb0 0x7f8c48032470 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f8c4c0096a0 tx=0x7f8c4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:17.065 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.064+0000 7f8c52ffd700 1 -- 172.21.15.178:0/3971375963 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8c580079c0 con 0x7f8c5c0f5410 2023-11-15T05:04:17.268 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.267+0000 7f8c646f8700 1 -- 172.21.15.178:0/3971375963 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c5c0fae60 con 0x7f8c5c0f5410 2023-11-15T05:04:17.269 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.268+0000 7f8c52ffd700 1 -- 172.21.15.178:0/3971375963 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8c5800f300 con 0x7f8c5c0f5410 2023-11-15T05:04:17.269 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:04:17.269 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:17.271 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.270+0000 7f8c646f8700 1 -- 172.21.15.178:0/3971375963 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8c4802ffb0 msgr2=0x7f8c48032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:17.271 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.270+0000 7f8c646f8700 1 --2- 172.21.15.178:0/3971375963 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8c4802ffb0 0x7f8c48032470 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f8c4c0096a0 tx=0x7f8c4c006b40).stop 2023-11-15T05:04:17.271 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.270+0000 7f8c646f8700 1 -- 172.21.15.178:0/3971375963 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c5c0f5410 msgr2=0x7f8c5c109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:17.271 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.270+0000 7f8c646f8700 1 --2- 172.21.15.178:0/3971375963 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c5c0f5410 0x7f8c5c109400 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f8c58014000 tx=0x7f8c5800b040).stop 2023-11-15T05:04:17.272 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.270+0000 7f8c646f8700 1 -- 172.21.15.178:0/3971375963 shutdown_connections 2023-11-15T05:04:17.272 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.270+0000 7f8c646f8700 1 --2- 172.21.15.178:0/3971375963 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f8c4802ffb0 0x7f8c48032470 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:17.272 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.270+0000 7f8c646f8700 1 --2- 172.21.15.178:0/3971375963 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c5c0f5410 0x7f8c5c109400 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:17.272 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.270+0000 7f8c646f8700 1 -- 172.21.15.178:0/3971375963 >> 172.21.15.178:0/3971375963 conn(0x7f8c5c0f0b70 msgr2=0x7f8c5c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:17.272 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.270+0000 7f8c646f8700 1 -- 172.21.15.178:0/3971375963 shutdown_connections 2023-11-15T05:04:17.273 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:17.270+0000 7f8c646f8700 1 -- 172.21.15.178:0/3971375963 wait complete. 2023-11-15T05:04:17.273 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:04:18.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:17 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:04:16.393959+0000 mgr.smithi163.qesyrx (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:18.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:17 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:04:17.268990+0000 mon.smithi163 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.178:0/3971375963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:18.682 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:18.683 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:19.840 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.839+0000 7fbaadf5d700 1 -- 172.21.15.178:0/701647194 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaa8060200 msgr2=0x7fbaa80fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:19.840 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.839+0000 7fbaadf5d700 1 --2- 172.21.15.178:0/701647194 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaa8060200 0x7fbaa80fb570 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fba90009fb0 tx=0x7fba90009b30).stop 2023-11-15T05:04:19.841 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.839+0000 7fbaadf5d700 1 -- 172.21.15.178:0/701647194 shutdown_connections 2023-11-15T05:04:19.841 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.839+0000 7fbaadf5d700 1 --2- 172.21.15.178:0/701647194 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaa8060200 0x7fbaa80fb570 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:19.841 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.839+0000 7fbaadf5d700 1 -- 172.21.15.178:0/701647194 >> 172.21.15.178:0/701647194 conn(0x7fbaa80f0b70 msgr2=0x7fbaa80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:19.841 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.839+0000 7fbaadf5d700 1 -- 172.21.15.178:0/701647194 shutdown_connections 2023-11-15T05:04:19.841 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.839+0000 7fbaadf5d700 1 -- 172.21.15.178:0/701647194 wait complete. 2023-11-15T05:04:19.841 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.840+0000 7fbaadf5d700 1 Processor -- start 2023-11-15T05:04:19.842 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.840+0000 7fbaadf5d700 1 -- start start 2023-11-15T05:04:19.842 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.840+0000 7fbaadf5d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaa8060200 0x7fbaa81088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:19.842 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.840+0000 7fbaadf5d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa8108dc0 con 0x7fbaa8060200 2023-11-15T05:04:19.844 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.840+0000 7fbaa77fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaa8060200 0x7fbaa81088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:19.844 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.840+0000 7fbaa77fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaa8060200 0x7fbaa81088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:47840/0 (socket says 172.21.15.178:47840) 2023-11-15T05:04:19.844 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.840+0000 7fbaa77fe700 1 -- 172.21.15.178:0/717271709 learned_addr learned my addr 172.21.15.178:0/717271709 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:04:19.844 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.841+0000 7fbaa77fe700 1 -- 172.21.15.178:0/717271709 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba90005040 con 0x7fbaa8060200 2023-11-15T05:04:19.845 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.841+0000 7fbaa77fe700 1 --2- 172.21.15.178:0/717271709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaa8060200 0x7fbaa81088d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fba90013510 tx=0x7fba9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:19.845 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.841+0000 7fba9ffff700 1 -- 172.21.15.178:0/717271709 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba90009320 con 0x7fbaa8060200 2023-11-15T05:04:19.845 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.841+0000 7fbaadf5d700 1 -- 172.21.15.178:0/717271709 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa8108fc0 con 0x7fbaa8060200 2023-11-15T05:04:19.845 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.841+0000 7fba9ffff700 1 -- 172.21.15.178:0/717271709 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba90013f00 con 0x7fbaa8060200 2023-11-15T05:04:19.845 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.841+0000 7fba9ffff700 1 -- 172.21.15.178:0/717271709 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba90014dc0 con 0x7fbaa8060200 2023-11-15T05:04:19.845 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.841+0000 7fbaadf5d700 1 -- 172.21.15.178:0/717271709 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa8109450 con 0x7fbaa8060200 2023-11-15T05:04:19.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.842+0000 7fba9ffff700 1 -- 172.21.15.178:0/717271709 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba90014420 con 0x7fbaa8060200 2023-11-15T05:04:19.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.842+0000 7fba9ffff700 1 --2- 172.21.15.178:0/717271709 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fba9402ff60 0x7fba94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:19.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.842+0000 7fbaadf5d700 1 -- 172.21.15.178:0/717271709 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa8045ad0 con 0x7fbaa8060200 2023-11-15T05:04:19.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.842+0000 7fba9ffff700 1 -- 172.21.15.178:0/717271709 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba900410d0 con 0x7fbaa8060200 2023-11-15T05:04:19.846 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.842+0000 7fbaa6ffd700 1 --2- 172.21.15.178:0/717271709 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fba9402ff60 0x7fba94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:19.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.842+0000 7fbaa6ffd700 1 --2- 172.21.15.178:0/717271709 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fba9402ff60 0x7fba94032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fba980096a0 tx=0x7fba98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:19.847 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:19.847+0000 7fba9ffff700 1 -- 172.21.15.178:0/717271709 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba90012330 con 0x7fbaa8060200 2023-11-15T05:04:20.053 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:20.053+0000 7fbaadf5d700 1 -- 172.21.15.178:0/717271709 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbaa810a300 con 0x7fbaa8060200 2023-11-15T05:04:20.054 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:20.053+0000 7fba9ffff700 1 -- 172.21.15.178:0/717271709 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fba9001cb10 con 0x7fbaa8060200 2023-11-15T05:04:20.054 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:04:20.054 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:20.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:20.055+0000 7fbaadf5d700 1 -- 172.21.15.178:0/717271709 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fba9402ff60 msgr2=0x7fba94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:20.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:20.055+0000 7fbaadf5d700 1 --2- 172.21.15.178:0/717271709 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fba9402ff60 0x7fba94032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fba980096a0 tx=0x7fba98006b40).stop 2023-11-15T05:04:20.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:20.055+0000 7fbaadf5d700 1 -- 172.21.15.178:0/717271709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaa8060200 msgr2=0x7fbaa81088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:20.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:20.055+0000 7fbaadf5d700 1 --2- 172.21.15.178:0/717271709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaa8060200 0x7fbaa81088d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fba90013510 tx=0x7fba9000b040).stop 2023-11-15T05:04:20.056 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:20.055+0000 7fbaadf5d700 1 -- 172.21.15.178:0/717271709 shutdown_connections 2023-11-15T05:04:20.057 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:20.055+0000 7fbaadf5d700 1 --2- 172.21.15.178:0/717271709 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fba9402ff60 0x7fba94032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:20.057 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:20.055+0000 7fbaadf5d700 1 --2- 172.21.15.178:0/717271709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbaa8060200 0x7fbaa81088d0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:20.058 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:20.055+0000 7fbaadf5d700 1 -- 172.21.15.178:0/717271709 >> 172.21.15.178:0/717271709 conn(0x7fbaa80f0b70 msgr2=0x7fbaa80f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:20.058 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:20.055+0000 7fbaadf5d700 1 -- 172.21.15.178:0/717271709 shutdown_connections 2023-11-15T05:04:20.059 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:20.055+0000 7fbaadf5d700 1 -- 172.21.15.178:0/717271709 wait complete. 2023-11-15T05:04:20.059 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:04:20.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:19 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:04:18.394265+0000 mgr.smithi163.qesyrx (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:21.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:20 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:04:20.054288+0000 mon.smithi163 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.178:0/717271709' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:21.453 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:21.454 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:22.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:21 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:04:20.394592+0000 mgr.smithi163.qesyrx (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:22.654 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.653+0000 7fa163260700 1 -- 172.21.15.178:0/2961388957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa15c0f6480 msgr2=0x7fa15c0f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:22.654 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.653+0000 7fa163260700 1 --2- 172.21.15.178:0/2961388957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa15c0f6480 0x7fa15c0f68a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fa150009fb0 tx=0x7fa150009b30).stop 2023-11-15T05:04:22.655 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.653+0000 7fa163260700 1 -- 172.21.15.178:0/2961388957 shutdown_connections 2023-11-15T05:04:22.655 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.653+0000 7fa163260700 1 --2- 172.21.15.178:0/2961388957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa15c0f6480 0x7fa15c0f68a0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:22.655 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.653+0000 7fa163260700 1 -- 172.21.15.178:0/2961388957 >> 172.21.15.178:0/2961388957 conn(0x7fa15c0f1c00 msgr2=0x7fa15c0f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:22.655 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.653+0000 7fa163260700 1 -- 172.21.15.178:0/2961388957 shutdown_connections 2023-11-15T05:04:22.656 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.653+0000 7fa163260700 1 -- 172.21.15.178:0/2961388957 wait complete. 2023-11-15T05:04:22.656 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.654+0000 7fa163260700 1 Processor -- start 2023-11-15T05:04:22.656 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.654+0000 7fa163260700 1 -- start start 2023-11-15T05:04:22.656 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.654+0000 7fa163260700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa15c0f6480 0x7fa15c1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:22.656 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.654+0000 7fa163260700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c106bc0 con 0x7fa15c0f6480 2023-11-15T05:04:22.658 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.654+0000 7fa160ffc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa15c0f6480 0x7fa15c1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:22.658 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.654+0000 7fa160ffc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa15c0f6480 0x7fa15c1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:47850/0 (socket says 172.21.15.178:47850) 2023-11-15T05:04:22.658 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.654+0000 7fa160ffc700 1 -- 172.21.15.178:0/1523770958 learned_addr learned my addr 172.21.15.178:0/1523770958 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:04:22.659 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.655+0000 7fa160ffc700 1 -- 172.21.15.178:0/1523770958 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa150005040 con 0x7fa15c0f6480 2023-11-15T05:04:22.659 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.655+0000 7fa160ffc700 1 --2- 172.21.15.178:0/1523770958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa15c0f6480 0x7fa15c1066d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fa150013510 tx=0x7fa15000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:22.659 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.655+0000 7fa1597fa700 1 -- 172.21.15.178:0/1523770958 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa150009320 con 0x7fa15c0f6480 2023-11-15T05:04:22.659 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.655+0000 7fa163260700 1 -- 172.21.15.178:0/1523770958 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa15c106dc0 con 0x7fa15c0f6480 2023-11-15T05:04:22.659 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.655+0000 7fa1597fa700 1 -- 172.21.15.178:0/1523770958 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa150013f00 con 0x7fa15c0f6480 2023-11-15T05:04:22.659 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.655+0000 7fa1597fa700 1 -- 172.21.15.178:0/1523770958 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa150014d90 con 0x7fa15c0f6480 2023-11-15T05:04:22.660 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.655+0000 7fa163260700 1 -- 172.21.15.178:0/1523770958 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa15c107250 con 0x7fa15c0f6480 2023-11-15T05:04:22.660 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.656+0000 7fa1597fa700 1 -- 172.21.15.178:0/1523770958 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa150014420 con 0x7fa15c0f6480 2023-11-15T05:04:22.661 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.656+0000 7fa163260700 1 -- 172.21.15.178:0/1523770958 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa15c045ad0 con 0x7fa15c0f6480 2023-11-15T05:04:22.661 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.656+0000 7fa1597fa700 1 --2- 172.21.15.178:0/1523770958 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa14c02ffb0 0x7fa14c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:22.661 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.656+0000 7fa1597fa700 1 -- 172.21.15.178:0/1523770958 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa150040fd0 con 0x7fa15c0f6480 2023-11-15T05:04:22.662 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.656+0000 7fa15bfff700 1 --2- 172.21.15.178:0/1523770958 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa14c02ffb0 0x7fa14c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:22.662 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.657+0000 7fa15bfff700 1 --2- 172.21.15.178:0/1523770958 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa14c02ffb0 0x7fa14c032470 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fa1480096a0 tx=0x7fa148006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:22.662 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.661+0000 7fa1597fa700 1 -- 172.21.15.178:0/1523770958 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa15006d2b0 con 0x7fa15c0f6480 2023-11-15T05:04:22.866 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.866+0000 7fa163260700 1 -- 172.21.15.178:0/1523770958 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa15c108100 con 0x7fa15c0f6480 2023-11-15T05:04:22.867 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.867+0000 7fa1597fa700 1 -- 172.21.15.178:0/1523770958 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa1500095f0 con 0x7fa15c0f6480 2023-11-15T05:04:22.868 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:04:22.868 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:22.869 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.868+0000 7fa163260700 1 -- 172.21.15.178:0/1523770958 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa14c02ffb0 msgr2=0x7fa14c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:22.869 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.869+0000 7fa163260700 1 --2- 172.21.15.178:0/1523770958 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa14c02ffb0 0x7fa14c032470 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fa1480096a0 tx=0x7fa148006b40).stop 2023-11-15T05:04:22.869 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.869+0000 7fa163260700 1 -- 172.21.15.178:0/1523770958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa15c0f6480 msgr2=0x7fa15c1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:22.870 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.869+0000 7fa163260700 1 --2- 172.21.15.178:0/1523770958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa15c0f6480 0x7fa15c1066d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fa150013510 tx=0x7fa15000b040).stop 2023-11-15T05:04:22.870 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.869+0000 7fa163260700 1 -- 172.21.15.178:0/1523770958 shutdown_connections 2023-11-15T05:04:22.870 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.869+0000 7fa163260700 1 --2- 172.21.15.178:0/1523770958 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fa14c02ffb0 0x7fa14c032470 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:22.870 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.869+0000 7fa163260700 1 --2- 172.21.15.178:0/1523770958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa15c0f6480 0x7fa15c1066d0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:22.871 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.869+0000 7fa163260700 1 -- 172.21.15.178:0/1523770958 >> 172.21.15.178:0/1523770958 conn(0x7fa15c0f1c00 msgr2=0x7fa15c0f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:22.871 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.869+0000 7fa163260700 1 -- 172.21.15.178:0/1523770958 shutdown_connections 2023-11-15T05:04:22.871 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:22.869+0000 7fa163260700 1 -- 172.21.15.178:0/1523770958 wait complete. 2023-11-15T05:04:22.871 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:04:23.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:22 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:04:22.867605+0000 mon.smithi163 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.178:0/1523770958' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:24.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:23 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:04:22.394860+0000 mgr.smithi163.qesyrx (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:24.256 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:24.257 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:25.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:24 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:04:24.457055+0000 mon.smithi163 (mon.0) 323 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:04:25.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:24 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:04:24.457501+0000 mon.smithi163 (mon.0) 324 : audit [INF] from='mgr.14162 172.21.15.163:0/4130725194' entity='mgr.smithi163.qesyrx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi163.qesyrx/trash_purge_schedule"}]: dispatch 2023-11-15T05:04:25.455 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.454+0000 7f1fe9fd3700 1 -- 172.21.15.178:0/1990711807 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fe4060280 msgr2=0x7f1fe40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:25.455 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.454+0000 7f1fe9fd3700 1 --2- 172.21.15.178:0/1990711807 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fe4060280 0x7f1fe40f90f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f1fcc009fb0 tx=0x7f1fcc009b30).stop 2023-11-15T05:04:25.455 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.454+0000 7f1fe9fd3700 1 -- 172.21.15.178:0/1990711807 shutdown_connections 2023-11-15T05:04:25.456 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.454+0000 7f1fe9fd3700 1 --2- 172.21.15.178:0/1990711807 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fe4060280 0x7f1fe40f90f0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:25.456 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.454+0000 7f1fe9fd3700 1 -- 172.21.15.178:0/1990711807 >> 172.21.15.178:0/1990711807 conn(0x7f1fe40f0b70 msgr2=0x7f1fe40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:25.456 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.454+0000 7f1fe9fd3700 1 -- 172.21.15.178:0/1990711807 shutdown_connections 2023-11-15T05:04:25.456 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.454+0000 7f1fe9fd3700 1 -- 172.21.15.178:0/1990711807 wait complete. 2023-11-15T05:04:25.457 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.455+0000 7f1fe9fd3700 1 Processor -- start 2023-11-15T05:04:25.459 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.455+0000 7f1fe9fd3700 1 -- start start 2023-11-15T05:04:25.459 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.455+0000 7f1fe9fd3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fe4060280 0x7f1fe41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:25.460 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.455+0000 7f1fe9fd3700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe4106bd0 con 0x7f1fe4060280 2023-11-15T05:04:25.460 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.455+0000 7f1fe37fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fe4060280 0x7f1fe41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:25.460 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.455+0000 7f1fe37fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fe4060280 0x7f1fe41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:57388/0 (socket says 172.21.15.178:57388) 2023-11-15T05:04:25.461 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.455+0000 7f1fe37fe700 1 -- 172.21.15.178:0/78285544 learned_addr learned my addr 172.21.15.178:0/78285544 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:04:25.461 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.456+0000 7f1fe37fe700 1 -- 172.21.15.178:0/78285544 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc005040 con 0x7f1fe4060280 2023-11-15T05:04:25.461 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.456+0000 7f1fe37fe700 1 --2- 172.21.15.178:0/78285544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fe4060280 0x7f1fe41066e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f1fcc013510 tx=0x7f1fcc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:25.461 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.456+0000 7f1fdbfff700 1 -- 172.21.15.178:0/78285544 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fcc009320 con 0x7f1fe4060280 2023-11-15T05:04:25.462 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.456+0000 7f1fe9fd3700 1 -- 172.21.15.178:0/78285544 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe4106dd0 con 0x7f1fe4060280 2023-11-15T05:04:25.462 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.456+0000 7f1fdbfff700 1 -- 172.21.15.178:0/78285544 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1fcc013f00 con 0x7f1fe4060280 2023-11-15T05:04:25.462 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.456+0000 7f1fdbfff700 1 -- 172.21.15.178:0/78285544 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fcc014dc0 con 0x7f1fe4060280 2023-11-15T05:04:25.462 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.456+0000 7f1fe9fd3700 1 -- 172.21.15.178:0/78285544 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe4107260 con 0x7f1fe4060280 2023-11-15T05:04:25.463 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.457+0000 7f1fdbfff700 1 -- 172.21.15.178:0/78285544 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1fcc0095f0 con 0x7f1fe4060280 2023-11-15T05:04:25.463 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.457+0000 7f1fe9fd3700 1 -- 172.21.15.178:0/78285544 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe4045ad0 con 0x7f1fe4060280 2023-11-15T05:04:25.463 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.457+0000 7f1fdbfff700 1 --2- 172.21.15.178:0/78285544 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1fd002ff60 0x7f1fd0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:25.463 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.457+0000 7f1fdbfff700 1 -- 172.21.15.178:0/78285544 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1fcc041040 con 0x7f1fe4060280 2023-11-15T05:04:25.463 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.457+0000 7f1fe2ffd700 1 --2- 172.21.15.178:0/78285544 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1fd002ff60 0x7f1fd0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:25.464 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.457+0000 7f1fe2ffd700 1 --2- 172.21.15.178:0/78285544 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1fd002ff60 0x7f1fd0032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f1fd40096a0 tx=0x7f1fd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:25.465 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.462+0000 7f1fdbfff700 1 -- 172.21.15.178:0/78285544 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1fcc014420 con 0x7f1fe4060280 2023-11-15T05:04:25.667 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.667+0000 7f1fe9fd3700 1 -- 172.21.15.178:0/78285544 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1fe4108110 con 0x7f1fe4060280 2023-11-15T05:04:25.668 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.667+0000 7f1fdbfff700 1 -- 172.21.15.178:0/78285544 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1fcc010230 con 0x7f1fe4060280 2023-11-15T05:04:25.668 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:04:25.668 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:25.669 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.669+0000 7f1fe9fd3700 1 -- 172.21.15.178:0/78285544 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1fd002ff60 msgr2=0x7f1fd0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:25.670 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.669+0000 7f1fe9fd3700 1 --2- 172.21.15.178:0/78285544 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1fd002ff60 0x7f1fd0032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f1fd40096a0 tx=0x7f1fd4006b40).stop 2023-11-15T05:04:25.670 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.669+0000 7f1fe9fd3700 1 -- 172.21.15.178:0/78285544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fe4060280 msgr2=0x7f1fe41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:25.670 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.669+0000 7f1fe9fd3700 1 --2- 172.21.15.178:0/78285544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fe4060280 0x7f1fe41066e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f1fcc013510 tx=0x7f1fcc00b040).stop 2023-11-15T05:04:25.670 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.669+0000 7f1fe9fd3700 1 -- 172.21.15.178:0/78285544 shutdown_connections 2023-11-15T05:04:25.670 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.669+0000 7f1fe9fd3700 1 --2- 172.21.15.178:0/78285544 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7f1fd002ff60 0x7f1fd0032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:25.671 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.669+0000 7f1fe9fd3700 1 --2- 172.21.15.178:0/78285544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fe4060280 0x7f1fe41066e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:25.671 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.669+0000 7f1fe9fd3700 1 -- 172.21.15.178:0/78285544 >> 172.21.15.178:0/78285544 conn(0x7f1fe40f0b70 msgr2=0x7f1fe40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:25.671 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.669+0000 7f1fe9fd3700 1 -- 172.21.15.178:0/78285544 shutdown_connections 2023-11-15T05:04:25.671 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:25.669+0000 7f1fe9fd3700 1 -- 172.21.15.178:0/78285544 wait complete. 2023-11-15T05:04:25.672 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:04:26.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:04:24.395200+0000 mgr.smithi163.qesyrx (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:26.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:25 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:04:25.668205+0000 mon.smithi163 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.178:0/78285544' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:27.066 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:27.066 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:28.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:27 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:04:26.395497+0000 mgr.smithi163.qesyrx (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:28.219 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.218+0000 7fc4c1948700 1 -- 172.21.15.178:0/4156040000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4bc060280 msgr2=0x7fc4bc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:28.219 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.218+0000 7fc4c1948700 1 --2- 172.21.15.178:0/4156040000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4bc060280 0x7fc4bc0f90f0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7fc4b0009fb0 tx=0x7fc4b0009b30).stop 2023-11-15T05:04:28.220 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.218+0000 7fc4c1948700 1 -- 172.21.15.178:0/4156040000 shutdown_connections 2023-11-15T05:04:28.220 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.218+0000 7fc4c1948700 1 --2- 172.21.15.178:0/4156040000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4bc060280 0x7fc4bc0f90f0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:28.220 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.218+0000 7fc4c1948700 1 -- 172.21.15.178:0/4156040000 >> 172.21.15.178:0/4156040000 conn(0x7fc4bc0f0b70 msgr2=0x7fc4bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:28.220 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.218+0000 7fc4c1948700 1 -- 172.21.15.178:0/4156040000 shutdown_connections 2023-11-15T05:04:28.220 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.218+0000 7fc4c1948700 1 -- 172.21.15.178:0/4156040000 wait complete. 2023-11-15T05:04:28.221 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.219+0000 7fc4c1948700 1 Processor -- start 2023-11-15T05:04:28.221 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.219+0000 7fc4c1948700 1 -- start start 2023-11-15T05:04:28.221 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.219+0000 7fc4c1948700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4bc060280 0x7fc4bc0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:28.221 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.219+0000 7fc4c1948700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc0f4ff0 con 0x7fc4bc060280 2023-11-15T05:04:28.223 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.219+0000 7fc4baffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4bc060280 0x7fc4bc0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:28.223 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.219+0000 7fc4baffd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4bc060280 0x7fc4bc0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:57402/0 (socket says 172.21.15.178:57402) 2023-11-15T05:04:28.223 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.219+0000 7fc4baffd700 1 -- 172.21.15.178:0/4126102203 learned_addr learned my addr 172.21.15.178:0/4126102203 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:04:28.224 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.219+0000 7fc4baffd700 1 -- 172.21.15.178:0/4126102203 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b0005040 con 0x7fc4bc060280 2023-11-15T05:04:28.224 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.220+0000 7fc4baffd700 1 --2- 172.21.15.178:0/4126102203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4bc060280 0x7fc4bc0f6b70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fc4b0013510 tx=0x7fc4b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:28.225 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.220+0000 7fc4a37fe700 1 -- 172.21.15.178:0/4126102203 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4b0009320 con 0x7fc4bc060280 2023-11-15T05:04:28.225 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.220+0000 7fc4c1948700 1 -- 172.21.15.178:0/4126102203 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4bc0f5190 con 0x7fc4bc060280 2023-11-15T05:04:28.225 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.220+0000 7fc4a37fe700 1 -- 172.21.15.178:0/4126102203 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc4b0013f00 con 0x7fc4bc060280 2023-11-15T05:04:28.225 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.220+0000 7fc4a37fe700 1 -- 172.21.15.178:0/4126102203 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4b0014dc0 con 0x7fc4bc060280 2023-11-15T05:04:28.225 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.220+0000 7fc4c1948700 1 -- 172.21.15.178:0/4126102203 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4bc0f5620 con 0x7fc4bc060280 2023-11-15T05:04:28.226 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.221+0000 7fc4a37fe700 1 -- 172.21.15.178:0/4126102203 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc4b00095f0 con 0x7fc4bc060280 2023-11-15T05:04:28.226 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.221+0000 7fc4c1948700 1 -- 172.21.15.178:0/4126102203 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4bc059100 con 0x7fc4bc060280 2023-11-15T05:04:28.226 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.221+0000 7fc4a37fe700 1 --2- 172.21.15.178:0/4126102203 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc4a402ff60 0x7fc4a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:28.226 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.221+0000 7fc4a37fe700 1 -- 172.21.15.178:0/4126102203 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc4b0041020 con 0x7fc4bc060280 2023-11-15T05:04:28.227 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.221+0000 7fc4ba7fc700 1 --2- 172.21.15.178:0/4126102203 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc4a402ff60 0x7fc4a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:28.227 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.221+0000 7fc4ba7fc700 1 --2- 172.21.15.178:0/4126102203 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc4a402ff60 0x7fc4a4032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fc4ac0096a0 tx=0x7fc4ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:28.227 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.226+0000 7fc4a37fe700 1 -- 172.21.15.178:0/4126102203 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc4b00263a0 con 0x7fc4bc060280 2023-11-15T05:04:28.431 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.430+0000 7fc4c1948700 1 -- 172.21.15.178:0/4126102203 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4bc045ad0 con 0x7fc4bc060280 2023-11-15T05:04:28.431 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.430+0000 7fc4a37fe700 1 -- 172.21.15.178:0/4126102203 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc4b001c920 con 0x7fc4bc060280 2023-11-15T05:04:28.431 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:04:28.432 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:28.433 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.432+0000 7fc4c1948700 1 -- 172.21.15.178:0/4126102203 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc4a402ff60 msgr2=0x7fc4a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:28.433 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.432+0000 7fc4c1948700 1 --2- 172.21.15.178:0/4126102203 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc4a402ff60 0x7fc4a4032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fc4ac0096a0 tx=0x7fc4ac006b40).stop 2023-11-15T05:04:28.433 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.432+0000 7fc4c1948700 1 -- 172.21.15.178:0/4126102203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4bc060280 msgr2=0x7fc4bc0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:28.433 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.432+0000 7fc4c1948700 1 --2- 172.21.15.178:0/4126102203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4bc060280 0x7fc4bc0f6b70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fc4b0013510 tx=0x7fc4b000b040).stop 2023-11-15T05:04:28.433 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.432+0000 7fc4c1948700 1 -- 172.21.15.178:0/4126102203 shutdown_connections 2023-11-15T05:04:28.434 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.432+0000 7fc4c1948700 1 --2- 172.21.15.178:0/4126102203 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fc4a402ff60 0x7fc4a4032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:28.434 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.432+0000 7fc4c1948700 1 --2- 172.21.15.178:0/4126102203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4bc060280 0x7fc4bc0f6b70 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:28.434 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.432+0000 7fc4c1948700 1 -- 172.21.15.178:0/4126102203 >> 172.21.15.178:0/4126102203 conn(0x7fc4bc0f0b70 msgr2=0x7fc4bc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:28.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.432+0000 7fc4c1948700 1 -- 172.21.15.178:0/4126102203 shutdown_connections 2023-11-15T05:04:28.435 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:28.432+0000 7fc4c1948700 1 -- 172.21.15.178:0/4126102203 wait complete. 2023-11-15T05:04:28.435 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:04:29.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:28 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: audit 2023-11-15T05:04:28.431092+0000 mon.smithi163 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.178:0/4126102203' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:29.865 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:29.866 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:30.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:29 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: cluster 2023-11-15T05:04:28.395840+0000 mgr.smithi163.qesyrx (mgr.14162) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:31.087 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.086+0000 7fe3a42ec700 1 -- 172.21.15.178:0/1573463142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe39c0f5410 msgr2=0x7fe39c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:31.088 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.086+0000 7fe3a42ec700 1 --2- 172.21.15.178:0/1573463142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe39c0f5410 0x7fe39c0f5830 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fe38c009fb0 tx=0x7fe38c009b30).stop 2023-11-15T05:04:31.088 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.087+0000 7fe3a42ec700 1 -- 172.21.15.178:0/1573463142 shutdown_connections 2023-11-15T05:04:31.088 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.087+0000 7fe3a42ec700 1 --2- 172.21.15.178:0/1573463142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe39c0f5410 0x7fe39c0f5830 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:31.088 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.087+0000 7fe3a42ec700 1 -- 172.21.15.178:0/1573463142 >> 172.21.15.178:0/1573463142 conn(0x7fe39c0f0b70 msgr2=0x7fe39c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:31.089 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.087+0000 7fe3a42ec700 1 -- 172.21.15.178:0/1573463142 shutdown_connections 2023-11-15T05:04:31.089 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.087+0000 7fe3a42ec700 1 -- 172.21.15.178:0/1573463142 wait complete. 2023-11-15T05:04:31.089 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.087+0000 7fe3a42ec700 1 Processor -- start 2023-11-15T05:04:31.089 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.087+0000 7fe3a42ec700 1 -- start start 2023-11-15T05:04:31.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.087+0000 7fe3a42ec700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe39c0f5410 0x7fe39c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:31.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.087+0000 7fe3a42ec700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c105b20 con 0x7fe39c0f5410 2023-11-15T05:04:31.093 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.087+0000 7fe3a2088700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe39c0f5410 0x7fe39c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:31.093 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.087+0000 7fe3a2088700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe39c0f5410 0x7fe39c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.178:57426/0 (socket says 172.21.15.178:57426) 2023-11-15T05:04:31.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.087+0000 7fe3a2088700 1 -- 172.21.15.178:0/11895679 learned_addr learned my addr 172.21.15.178:0/11895679 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-11-15T05:04:31.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.088+0000 7fe3a2088700 1 -- 172.21.15.178:0/11895679 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38c005040 con 0x7fe39c0f5410 2023-11-15T05:04:31.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.088+0000 7fe3a2088700 1 --2- 172.21.15.178:0/11895679 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe39c0f5410 0x7fe39c105630 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fe38c013510 tx=0x7fe38c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:31.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.088+0000 7fe392ffd700 1 -- 172.21.15.178:0/11895679 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe38c009320 con 0x7fe39c0f5410 2023-11-15T05:04:31.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.088+0000 7fe392ffd700 1 -- 172.21.15.178:0/11895679 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe38c013f00 con 0x7fe39c0f5410 2023-11-15T05:04:31.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.088+0000 7fe3a42ec700 1 -- 172.21.15.178:0/11895679 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe39c105d20 con 0x7fe39c0f5410 2023-11-15T05:04:31.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.088+0000 7fe392ffd700 1 -- 172.21.15.178:0/11895679 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe38c014d90 con 0x7fe39c0f5410 2023-11-15T05:04:31.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.088+0000 7fe3a42ec700 1 -- 172.21.15.178:0/11895679 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39c1061b0 con 0x7fe39c0f5410 2023-11-15T05:04:31.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.089+0000 7fe392ffd700 1 -- 172.21.15.178:0/11895679 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe38c0095f0 con 0x7fe39c0f5410 2023-11-15T05:04:31.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.089+0000 7fe3a42ec700 1 -- 172.21.15.178:0/11895679 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe39c045ad0 con 0x7fe39c0f5410 2023-11-15T05:04:31.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.089+0000 7fe392ffd700 1 --2- 172.21.15.178:0/11895679 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe38802ff60 0x7fe388032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:31.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.089+0000 7fe392ffd700 1 -- 172.21.15.178:0/11895679 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe38c041000 con 0x7fe39c0f5410 2023-11-15T05:04:31.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.089+0000 7fe3a1887700 1 --2- 172.21.15.178:0/11895679 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe38802ff60 0x7fe388032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:31.097 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.090+0000 7fe3a1887700 1 --2- 172.21.15.178:0/11895679 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe38802ff60 0x7fe388032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fe3980096a0 tx=0x7fe398006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:31.098 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.094+0000 7fe392ffd700 1 -- 172.21.15.178:0/11895679 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe38c014420 con 0x7fe39c0f5410 2023-11-15T05:04:31.299 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.299+0000 7fe3a42ec700 1 -- 172.21.15.178:0/11895679 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe39c0fae60 con 0x7fe39c0f5410 2023-11-15T05:04:31.300 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.300+0000 7fe392ffd700 1 -- 172.21.15.178:0/11895679 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe38c01d7f0 con 0x7fe39c0f5410 2023-11-15T05:04:31.301 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-11-15T05:04:31.301 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"01689f7e-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T04:54:31.569260Z","created":"2023-11-15T04:54:31.569260Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:31.302 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.301+0000 7fe3a42ec700 1 -- 172.21.15.178:0/11895679 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe38802ff60 msgr2=0x7fe388032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:31.302 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.302+0000 7fe3a42ec700 1 --2- 172.21.15.178:0/11895679 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe38802ff60 0x7fe388032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fe3980096a0 tx=0x7fe398006b40).stop 2023-11-15T05:04:31.302 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.302+0000 7fe3a42ec700 1 -- 172.21.15.178:0/11895679 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe39c0f5410 msgr2=0x7fe39c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:31.303 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.302+0000 7fe3a42ec700 1 --2- 172.21.15.178:0/11895679 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe39c0f5410 0x7fe39c105630 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fe38c013510 tx=0x7fe38c00b040).stop 2023-11-15T05:04:31.303 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.302+0000 7fe3a42ec700 1 -- 172.21.15.178:0/11895679 shutdown_connections 2023-11-15T05:04:31.303 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.302+0000 7fe3a42ec700 1 --2- 172.21.15.178:0/11895679 >> [v2:172.21.15.163:6800/3399151648,v1:172.21.15.163:6801/3399151648] conn(0x7fe38802ff60 0x7fe388032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:31.303 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.302+0000 7fe3a42ec700 1 --2- 172.21.15.178:0/11895679 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe39c0f5410 0x7fe39c105630 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:31.304 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.302+0000 7fe3a42ec700 1 -- 172.21.15.178:0/11895679 >> 172.21.15.178:0/11895679 conn(0x7fe39c0f0b70 msgr2=0x7fe39c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:31.304 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.302+0000 7fe3a42ec700 1 -- 172.21.15.178:0/11895679 shutdown_connections 2023-11-15T05:04:31.304 INFO:teuthology.orchestra.run.smithi178.stderr:2023-11-15T05:04:31.302+0000 7fe3a42ec700 1 -- 172.21.15.178:0/11895679 wait complete. 2023-11-15T05:04:31.304 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-11-15T05:04:31.697 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2023-11-15T05:04:31.697 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-11-15T05:04:31.698 DEBUG:teuthology.orchestra.run.smithi163:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-15T05:04:31.723 DEBUG:teuthology.orchestra.run.smithi178:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-15T05:04:31.745 INFO:tasks.cephadm:Stopping all daemons... 2023-11-15T05:04:31.745 INFO:tasks.cephadm.mon.smithi163:Stopping mon.smithi163... 2023-11-15T05:04:31.746 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mon.smithi163 2023-11-15T05:04:32.051 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:31 smithi163 systemd[1]: Stopping Ceph mon.smithi163 for 01689f7e-8373-11ee-8dbe-212e2dc638e7... 2023-11-15T05:04:32.051 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:31 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T05:04:31.865+0000 7f60df172700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi163 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-15T05:04:32.052 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Nov 15 05:04:31 smithi163 ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7-mon.smithi163[112345]: debug 2023-11-15T05:04:31.865+0000 7f60df172700 -1 mon.smithi163@0(leader) e1 *** Got Signal Terminated *** 2023-11-15T05:04:32.177 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mon.smithi163.service' 2023-11-15T05:04:32.234 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-15T05:04:32.235 INFO:tasks.cephadm.mon.smithi163:Stopped mon.smithi163 2023-11-15T05:04:32.236 INFO:tasks.cephadm.mon.smithi178:Stopping mon.smithi178... 2023-11-15T05:04:32.236 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mon.smithi178 2023-11-15T05:04:32.269 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mon.smithi178.service: Unit ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mon.smithi178.service not loaded. 2023-11-15T05:04:32.271 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-11-15T05:04:32.272 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi178" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi178 with status 5: 'sudo systemctl stop ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mon.smithi178' 2023-11-15T05:04:32.272 INFO:tasks.cephadm:Archiving crash dumps... 2023-11-15T05:04:32.275 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458408/remote/ubuntu@smithi163.front.sepia.ceph.com/crash 2023-11-15T05:04:32.276 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-15T05:04:32.311 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458408/remote/ubuntu@smithi178.front.sepia.ceph.com/crash 2023-11-15T05:04:32.312 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-15T05:04:32.350 INFO:teuthology.orchestra.run.smithi178.stderr:tar: /var/lib/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-15T05:04:32.351 INFO:teuthology.orchestra.run.smithi178.stderr:tar: Error is not recoverable: exiting now 2023-11-15T05:04:32.352 INFO:tasks.cephadm:Checking cluster log for badness... 2023-11-15T05:04:32.353 DEBUG:teuthology.orchestra.run.smithi163:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-11-15T05:04:32.383 INFO:teuthology.orchestra.run.smithi163.stderr:grep: /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/ceph.log: No such file or directory 2023-11-15T05:04:32.384 INFO:tasks.cephadm:Compressing logs... 2023-11-15T05:04:32.385 DEBUG:teuthology.orchestra.run.smithi163:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-15T05:04:32.429 DEBUG:teuthology.orchestra.run.smithi178:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-15T05:04:32.454 INFO:teuthology.orchestra.run.smithi163.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-15T05:04:32.454 INFO:teuthology.orchestra.run.smithi163.stderr:: No such file or directory 2023-11-15T05:04:32.454 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/ceph-volume.log 2023-11-15T05:04:32.455 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/ceph-mon.smithi163.log 2023-11-15T05:04:32.456 INFO:teuthology.orchestra.run.smithi178.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- : No such file or directory/var/log/ceph/cephadm.log 2023-11-15T05:04:32.456 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-15T05:04:32.457 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi163.qesyrx.log 2023-11-15T05:04:32.457 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/ceph-volume.log: /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/ceph-mon.smithi163.log: /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi163.qesyrx.log: 90.8% -- replaced with /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/ceph-volume.log.gz 2023-11-15T05:04:32.457 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/cephadm.log: 96.2% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-15T05:04:32.459 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-11-15T05:04:32.459 INFO:teuthology.orchestra.run.smithi178.stderr:real 0m0.013s 2023-11-15T05:04:32.461 INFO:teuthology.orchestra.run.smithi178.stderr:user 0m0.009s 2023-11-15T05:04:32.461 INFO:teuthology.orchestra.run.smithi178.stderr:sys 0m0.011s 2023-11-15T05:04:32.462 INFO:teuthology.orchestra.run.smithi163.stderr: 92.9% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-15T05:04:32.531 INFO:teuthology.orchestra.run.smithi163.stderr:gzip: /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi163.qesyrx.log: file size changed while zipping 2023-11-15T05:04:32.532 INFO:teuthology.orchestra.run.smithi163.stderr: 92.6% -- replaced with /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi163.qesyrx.log.gz 2023-11-15T05:04:32.597 INFO:teuthology.orchestra.run.smithi163.stderr: 90.5% -- replaced with /var/log/ceph/01689f7e-8373-11ee-8dbe-212e2dc638e7/ceph-mon.smithi163.log.gz 2023-11-15T05:04:32.599 INFO:teuthology.orchestra.run.smithi163.stderr: 2023-11-15T05:04:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:real 0m0.155s 2023-11-15T05:04:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:user 0m0.230s 2023-11-15T05:04:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:sys 0m0.018s 2023-11-15T05:04:32.601 INFO:tasks.cephadm:Archiving logs... 2023-11-15T05:04:32.602 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458408/remote/ubuntu@smithi163.front.sepia.ceph.com/log 2023-11-15T05:04:32.603 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-15T05:04:32.717 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458408/remote/ubuntu@smithi178.front.sepia.ceph.com/log 2023-11-15T05:04:32.718 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-15T05:04:32.751 INFO:tasks.cephadm:Removing cluster... 2023-11-15T05:04:32.752 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 --force 2023-11-15T05:06:03.149 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 01689f7e-8373-11ee-8dbe-212e2dc638e7 --force 2023-11-15T05:06:03.456 INFO:tasks.cephadm:Removing cephadm ... 2023-11-15T05:06:03.457 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-15T05:06:03.476 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-15T05:06:03.495 INFO:tasks.cephadm:Teardown complete 2023-11-15T05:06:03.495 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi178 with status 5: 'sudo systemctl stop ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mon.smithi178' 2023-11-15T05:06:03.583 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d1aa09fbbb6d42c9a6ae612434639bf9 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi178 with status 5: 'sudo systemctl stop ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mon.smithi178' 2023-11-15T05:06:03.589 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-11-15T05:06:03.598 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-15T05:06:03.613 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-15T05:06:03.614 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-15T05:06:03.646 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-15T05:06:03.728 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-15T05:06:03.738 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-15T05:06:03.739 DEBUG:teuthology.orchestra.run.smithi163:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T05:06:03.741 DEBUG:teuthology.orchestra.run.smithi178:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T05:06:03.756 INFO:teuthology.orchestra.run.smithi163.stderr:bash: ntpq: command not found 2023-11-15T05:06:03.758 INFO:teuthology.orchestra.run.smithi178.stderr:bash: ntpq: command not found 2023-11-15T05:06:03.759 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T05:06:03.760 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2023-11-15T05:06:03.760 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 +37us[ +37us] +/- 95ms 2023-11-15T05:06:03.760 INFO:teuthology.orchestra.run.smithi163.stdout:^* hv02.front.sepia.ceph.com 3 6 377 29 -209us[ -209us] +/- 49ms 2023-11-15T05:06:03.761 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +301us[ +301us] +/- 88ms 2023-11-15T05:06:03.761 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T05:06:03.762 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T05:06:03.763 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2023-11-15T05:06:03.763 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 27 +32us[ +32us] +/- 95ms 2023-11-15T05:06:03.763 INFO:teuthology.orchestra.run.smithi178.stdout:^* hv02.front.sepia.ceph.com 3 6 377 28 -201us[ -216us] +/- 49ms 2023-11-15T05:06:03.764 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 +304us[ +290us] +/- 88ms 2023-11-15T05:06:03.764 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T05:06:03.765 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-15T05:06:03.777 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-15T05:06:03.778 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-15T05:06:03.789 DEBUG:teuthology.orchestra.run.smithi163:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-15T05:06:03.889 DEBUG:teuthology.orchestra.run.smithi178:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-15T05:06:03.978 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T05:06:04.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T05:06:04.011 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T05:06:04.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T05:06:04.044 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-15T05:06:04.065 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-15T05:06:04.154 INFO:teuthology.task.internal:Duration was 1392.931386 seconds 2023-11-15T05:06:04.159 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-15T05:06:04.369 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-15T05:06:04.369 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-15T05:06:04.372 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-15T05:06:04.417 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T05:06:04.418 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T05:06:04.895 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-15T05:06:04.895 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com 2023-11-15T05:06:04.896 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-15T05:06:04.921 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi178.front.sepia.ceph.com 2023-11-15T05:06:04.922 DEBUG:teuthology.orchestra.run.smithi178:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-15T05:06:04.946 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-15T05:06:04.947 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-15T05:06:04.964 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-15T05:06:05.033 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-15T05:06:05.034 DEBUG:teuthology.orchestra.run.smithi163:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-15T05:06:05.036 DEBUG:teuthology.orchestra.run.smithi178:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-15T05:06:05.125 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-15T05:06:05.136 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-15T05:06:05.137 DEBUG:teuthology.orchestra.run.smithi163:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-15T05:06:05.164 DEBUG:teuthology.orchestra.run.smithi178:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-15T05:06:05.191 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-15T05:06:05.205 DEBUG:teuthology.orchestra.run.smithi163:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-15T05:06:05.211 DEBUG:teuthology.orchestra.run.smithi178:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-15T05:06:05.232 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = core 2023-11-15T05:06:05.257 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = core 2023-11-15T05:06:05.276 DEBUG:teuthology.orchestra.run.smithi163:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-15T05:06:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T05:06:05.323 DEBUG:teuthology.orchestra.run.smithi178:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-15T05:06:05.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T05:06:05.341 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-15T05:06:05.352 INFO:teuthology.task.internal:Transferring archived files... 2023-11-15T05:06:05.353 DEBUG:teuthology.misc:Transferring archived files from smithi163:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458408/remote/smithi163 2023-11-15T05:06:05.354 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-15T05:06:05.404 DEBUG:teuthology.misc:Transferring archived files from smithi178:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458408/remote/smithi178 2023-11-15T05:06:05.405 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-15T05:06:05.448 INFO:teuthology.task.internal:Removing archive directory... 2023-11-15T05:06:05.448 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-15T05:06:05.451 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-15T05:06:05.498 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-15T05:06:05.511 INFO:teuthology.task.internal:Not uploading archives. 2023-11-15T05:06:05.511 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-15T05:06:05.521 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-15T05:06:05.522 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-15T05:06:05.525 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-15T05:06:05.540 INFO:teuthology.orchestra.run.smithi163.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 15 05:06 /home/ubuntu/cephtest 2023-11-15T05:06:05.556 INFO:teuthology.orchestra.run.smithi178.stdout: 393225 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 15 05:06 /home/ubuntu/cephtest 2023-11-15T05:06:05.557 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-15T05:06:05.569 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-15T05:06:05.594 INFO:teuthology.nuke:Checking targets against current locks 2023-11-15T05:06:05.628 DEBUG:teuthology.nuke:shortname: smithi163 2023-11-15T05:06:05.628 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T05:06:05.655 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458408', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 04:32:51.442054', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMbZUy/XzIHNhYEhMk6LE/09PqylObCPG64qcQlr3f87vYGqpGiLHv8nRzcVuIw3HMujdi66qU5CTxoX5cQFf50='} 2023-11-15T05:06:05.660 DEBUG:teuthology.nuke:shortname: smithi178 2023-11-15T05:06:05.660 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T05:06:05.685 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458408', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 04:32:51.443506', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNDbgYmv5wQAliGAJtcf6okb0phTjo29iPGnXDhDWh3Du70JD0vxr/tvfeVe/JRQu2Iacuq5Qf3GdbMyIlB8wmA='} 2023-11-15T05:06:05.719 INFO:teuthology.orchestra.console.smithi163:Power off 2023-11-15T05:06:05.720 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-15T05:06:05.751 DEBUG:teuthology.orchestra.console.smithi163:power off output: Chassis Power Control: Down/Off 2023-11-15T05:06:05.751 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T05:06:05.769 INFO:teuthology.orchestra.console.smithi178:Power off 2023-11-15T05:06:05.770 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-15T05:06:05.783 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2023-11-15T05:06:05.804 DEBUG:teuthology.orchestra.console.smithi178:power off output: Chassis Power Control: Down/Off 2023-11-15T05:06:05.805 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T05:06:05.841 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2023-11-15T05:06:09.784 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T05:06:09.844 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T05:06:09.897 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2023-11-15T05:06:09.955 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2023-11-15T05:06:13.902 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T05:06:13.956 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T05:06:14.022 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2023-11-15T05:06:14.072 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is off 2023-11-15T05:06:14.173 INFO:teuthology.orchestra.console.smithi178:Power off completed 2023-11-15T05:06:18.026 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T05:06:18.137 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is off 2023-11-15T05:06:18.238 INFO:teuthology.orchestra.console.smithi163:Power off completed 2023-11-15T05:06:18.346 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1392.9313859939575 failure_reason: 'Command failed on smithi178 with status 5: ''sudo systemctl stop ceph-01689f7e-8373-11ee-8dbe-212e2dc638e7@mon.smithi178''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=d1aa09fbbb6d42c9a6ae612434639bf9 status: fail success: false 2023-11-15T05:06:18.347 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-15T05:06:18.429 INFO:teuthology.run:FAIL