2024-01-21T19:34:42.819 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-21T19:34:42.819 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --archive /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525059 --owner scheduled_yuriw@teuthology --verbose --name yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525059/orig.config.yaml 2024-01-21T19:34:42.868 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-21T19:34:42.962 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525059 branch: wip-yuri4-testing-2024-01-18-1257-pacific description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: null first_in_suite: false job_id: '7525059' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2024-01-18-1257-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2024-01-18-1257-pacific sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 9457 sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-yuri4-testing-2024-01-18-1257-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec targets: smithi169.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPebK3S0muhRwMXStynRW0aoWwxHFF3sjrD/oODnl7qYpB7hdQsoZW1TKv1Ezb4tJZy16hfjIJBJUL6oPdvScpo= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ3RlVg3usAaY8CQmD/14tkcGNhuOOBr183OX1FthtbBppgoH2/4EHU2ec/NFzXx791hRiW0c+lfxobzK60ZI2w= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-21_16:37:18 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-01-21T19:34:42.962 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa; will attempt to use it 2024-01-21T19:34:42.963 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks 2024-01-21T19:34:42.963 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-21T19:34:42.965 INFO:teuthology.task.internal:Checking packages... 2024-01-21T19:34:42.988 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec' 2024-01-21T19:34:42.988 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-21T19:34:42.989 INFO:teuthology.packaging:ref: None 2024-01-21T19:34:42.989 INFO:teuthology.packaging:tag: None 2024-01-21T19:34:42.989 INFO:teuthology.packaging:branch: wip-yuri4-testing-2024-01-18-1257-pacific 2024-01-21T19:34:42.989 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T19:34:42.989 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2024-01-18-1257-pacific 2024-01-21T19:34:43.211 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-454.gea310589 2024-01-21T19:34:43.213 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-21T19:34:43.219 INFO:teuthology.task.internal:no buildpackages task found 2024-01-21T19:34:43.219 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-21T19:34:43.228 INFO:teuthology.task.internal:Saving configuration 2024-01-21T19:34:43.246 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-21T19:34:43.252 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T19:34:43.277 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525059', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 19:29:25.919598', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPebK3S0muhRwMXStynRW0aoWwxHFF3sjrD/oODnl7qYpB7hdQsoZW1TKv1Ezb4tJZy16hfjIJBJUL6oPdvScpo='} 2024-01-21T19:34:43.302 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525059', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 19:29:25.921557', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ3RlVg3usAaY8CQmD/14tkcGNhuOOBr183OX1FthtbBppgoH2/4EHU2ec/NFzXx791hRiW0c+lfxobzK60ZI2w='} 2024-01-21T19:34:43.302 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-21T19:34:43.308 INFO:teuthology.task.internal:roles: ubuntu@smithi169.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-01-21T19:34:43.309 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-01-21T19:34:43.309 INFO:teuthology.run_tasks:Running task console_log... 2024-01-21T19:34:43.382 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5531eb5160>, signals=[15]) 2024-01-21T19:34:43.382 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-21T19:34:43.388 INFO:teuthology.task.internal:Opening connections... 2024-01-21T19:34:43.389 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi169.front.sepia.ceph.com 2024-01-21T19:34:43.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:34:43.471 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-01-21T19:34:43.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:34:43.544 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-21T19:34:43.551 DEBUG:teuthology.orchestra.run.smithi169:> uname -m 2024-01-21T19:34:43.573 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64 2024-01-21T19:34:43.574 DEBUG:teuthology.orchestra.run.smithi169:> cat /etc/os-release 2024-01-21T19:34:43.635 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="CentOS Stream" 2024-01-21T19:34:43.635 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="8" 2024-01-21T19:34:43.635 INFO:teuthology.orchestra.run.smithi169.stdout:ID="centos" 2024-01-21T19:34:43.635 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE="rhel fedora" 2024-01-21T19:34:43.635 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="8" 2024-01-21T19:34:43.635 INFO:teuthology.orchestra.run.smithi169.stdout:PLATFORM_ID="platform:el8" 2024-01-21T19:34:43.635 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-21T19:34:43.635 INFO:teuthology.orchestra.run.smithi169.stdout:ANSI_COLOR="0;31" 2024-01-21T19:34:43.635 INFO:teuthology.orchestra.run.smithi169.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-21T19:34:43.636 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://centos.org/" 2024-01-21T19:34:43.636 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-21T19:34:43.636 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-21T19:34:43.636 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-21T19:34:43.636 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2024-01-21T19:34:43.655 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-01-21T19:34:43.677 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-01-21T19:34:43.678 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-01-21T19:34:43.739 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-01-21T19:34:43.740 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="8" 2024-01-21T19:34:43.740 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-01-21T19:34:43.740 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-01-21T19:34:43.740 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="8" 2024-01-21T19:34:43.740 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el8" 2024-01-21T19:34:43.740 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-21T19:34:43.740 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-01-21T19:34:43.740 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-21T19:34:43.740 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-01-21T19:34:43.740 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-21T19:34:43.740 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-21T19:34:43.741 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-21T19:34:43.741 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-01-21T19:34:43.764 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-21T19:34:43.774 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-21T19:34:43.817 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-21T19:34:43.817 DEBUG:teuthology.orchestra.run.smithi169:> test '!' -e /home/ubuntu/cephtest 2024-01-21T19:34:43.820 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-01-21T19:34:43.840 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-21T19:34:43.848 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-21T19:34:43.848 DEBUG:teuthology.orchestra.run.smithi169:> test -z $(ls -A /var/lib/ceph) 2024-01-21T19:34:43.882 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-01-21T19:34:43.926 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-21T19:34:43.984 INFO:teuthology.run_tasks:Running task kernel... 2024-01-21T19:34:44.001 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-21T19:34:44.002 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-21T19:34:44.002 DEBUG:teuthology.orchestra.run.smithi169:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-21T19:34:44.003 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-21T19:34:44.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T19:34:44.024 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-01-21T19:34:44.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T19:34:44.025 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-01-21T19:34:44.085 INFO:teuthology.orchestra.run.smithi173.stdout:4.18.0-489.el8.x86_64 2024-01-21T19:34:44.085 INFO:teuthology.task.kernel:Running kernel on smithi173: 4.18.0-489.el8.x86_64 2024-01-21T19:34:44.085 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum install -y kernel 2024-01-21T19:34:44.086 INFO:teuthology.orchestra.run.smithi169.stdout:4.18.0-489.el8.x86_64 2024-01-21T19:34:44.087 INFO:teuthology.task.kernel:Running kernel on smithi169: 4.18.0-489.el8.x86_64 2024-01-21T19:34:44.087 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum install -y kernel 2024-01-21T19:34:48.000 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 8 - AppStream 19 MB/s | 26 MB 00:01 2024-01-21T19:34:48.570 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - AppStream 15 MB/s | 26 MB 00:01 2024-01-21T19:34:53.592 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 8 - BaseOS 12 MB/s | 10 MB 00:00 2024-01-21T19:34:53.987 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - BaseOS 14 MB/s | 10 MB 00:00 2024-01-21T19:34:55.725 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 8 - Extras 32 kB/s | 18 kB 00:00 2024-01-21T19:34:55.979 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - Extras 45 kB/s | 18 kB 00:00 2024-01-21T19:34:56.051 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 8 - Extras common packages 32 kB/s | 6.9 kB 00:00 2024-01-21T19:34:56.302 INFO:teuthology.orchestra.run.smithi169.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-01-21T19:34:56.329 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - Extras common packages 29 kB/s | 6.9 kB 00:00 2024-01-21T19:34:56.581 INFO:teuthology.orchestra.run.smithi173.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-01-21T19:34:57.797 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-01-21T19:34:57.909 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2024-01-21T19:35:01.578 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 401 kB/s | 24 kB 00:00 2024-01-21T19:35:01.632 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 397 kB/s | 24 kB 00:00 2024-01-21T19:35:04.483 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-21T19:35:04.483 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-21T19:35:04.504 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-21T19:35:04.504 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-21T19:35:04.602 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:35:04.602 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:35:04.603 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-01-21T19:35:04.603 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:35:04.603 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-21T19:35:04.603 INFO:teuthology.orchestra.run.smithi169.stdout: kernel x86_64 4.18.0-535.el8 baseos 10 M 2024-01-21T19:35:04.603 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-01-21T19:35:04.603 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core x86_64 4.18.0-535.el8 baseos 43 M 2024-01-21T19:35:04.603 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules x86_64 4.18.0-535.el8 baseos 36 M 2024-01-21T19:35:04.603 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:35:04.603 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-21T19:35:04.603 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:35:04.603 INFO:teuthology.orchestra.run.smithi169.stdout:Install 3 Packages 2024-01-21T19:35:04.603 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:35:04.604 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 89 M 2024-01-21T19:35:04.604 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 96 M 2024-01-21T19:35:04.604 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-21T19:35:04.619 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:35:04.620 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:35:04.620 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-01-21T19:35:04.620 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:35:04.620 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-01-21T19:35:04.621 INFO:teuthology.orchestra.run.smithi173.stdout: kernel x86_64 4.18.0-535.el8 baseos 10 M 2024-01-21T19:35:04.621 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-01-21T19:35:04.621 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core x86_64 4.18.0-535.el8 baseos 43 M 2024-01-21T19:35:04.621 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules x86_64 4.18.0-535.el8 baseos 36 M 2024-01-21T19:35:04.621 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:35:04.621 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-01-21T19:35:04.621 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:35:04.621 INFO:teuthology.orchestra.run.smithi173.stdout:Install 3 Packages 2024-01-21T19:35:04.621 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:35:04.622 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 89 M 2024-01-21T19:35:04.622 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 96 M 2024-01-21T19:35:04.622 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-01-21T19:35:05.929 INFO:teuthology.orchestra.run.smithi169.stdout:(1/3): kernel-4.18.0-535.el8.x86_64.rpm 14 MB/s | 10 MB 00:00 2024-01-21T19:35:06.900 INFO:teuthology.orchestra.run.smithi173.stdout:(1/3): kernel-4.18.0-535.el8.x86_64.rpm 9.6 MB/s | 10 MB 00:01 2024-01-21T19:35:07.079 INFO:teuthology.orchestra.run.smithi169.stdout:(2/3): kernel-modules-4.18.0-535.el8.x86_64.rpm 19 MB/s | 36 MB 00:01 2024-01-21T19:35:07.688 INFO:teuthology.orchestra.run.smithi169.stdout:(3/3): kernel-core-4.18.0-535.el8.x86_64.rpm 17 MB/s | 43 MB 00:02 2024-01-21T19:35:07.688 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:35:07.689 INFO:teuthology.orchestra.run.smithi169.stdout:Total 29 MB/s | 89 MB 00:03 2024-01-21T19:35:08.100 INFO:teuthology.orchestra.run.smithi173.stdout:(2/3): kernel-modules-4.18.0-535.el8.x86_64.rpm 16 MB/s | 36 MB 00:02 2024-01-21T19:35:08.368 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-21T19:35:08.717 INFO:teuthology.orchestra.run.smithi173.stdout:(3/3): kernel-core-4.18.0-535.el8.x86_64.rpm 15 MB/s | 43 MB 00:02 2024-01-21T19:35:08.717 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:35:08.717 INFO:teuthology.orchestra.run.smithi173.stdout:Total 22 MB/s | 89 MB 00:04 2024-01-21T19:35:08.801 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-21T19:35:08.801 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-21T19:35:09.393 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-01-21T19:35:09.805 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-01-21T19:35:09.805 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-01-21T19:35:10.037 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-21T19:35:10.070 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-21T19:35:11.044 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-01-21T19:35:11.077 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-01-21T19:35:12.830 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-21T19:35:13.293 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T19:35:13.860 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-01-21T19:35:14.411 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T19:35:16.867 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T19:35:17.251 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T19:35:17.980 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T19:35:18.427 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T19:35:20.681 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T19:35:20.956 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:35:21.851 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T19:35:22.142 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:35:42.792 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:35:42.817 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:35:43.162 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:35:43.162 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-4.18.0-535.el8.x86_64 1/3 2024-01-21T19:35:43.162 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-core-4.18.0-535.el8.x86_64 2/3 2024-01-21T19:35:43.676 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:35:43.676 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:35:43.676 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-21T19:35:43.676 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-4.18.0-535.el8.x86_64 kernel-core-4.18.0-535.el8.x86_64 2024-01-21T19:35:43.676 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-4.18.0-535.el8.x86_64 2024-01-21T19:35:43.677 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:35:43.677 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:35:43.822 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T19:35:44.079 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T19:35:44.080 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T19:35:44.080 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-535.el8.x86_64 2024-01-21T19:35:44.080 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-21T19:35:44.080 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-21T19:35:44.080 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-01-21T19:35:44.080 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi169.front.sepia.ceph.com, path=None, version=distro) 2024-01-21T19:35:44.080 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T19:35:44.147 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:35:44.170 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:35:44.290 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T19:35:44.290 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T19:35:44.291 DEBUG:teuthology.orchestra.run.smithi169:> sudo rpm -qi grub2-tools 2024-01-21T19:35:44.324 INFO:teuthology.orchestra.run.smithi169.stdout:Name : grub2-tools 2024-01-21T19:35:44.324 INFO:teuthology.orchestra.run.smithi169.stdout:Epoch : 1 2024-01-21T19:35:44.324 INFO:teuthology.orchestra.run.smithi169.stdout:Version : 2.02 2024-01-21T19:35:44.324 INFO:teuthology.orchestra.run.smithi169.stdout:Release : 129.el8 2024-01-21T19:35:44.324 INFO:teuthology.orchestra.run.smithi169.stdout:Architecture: x86_64 2024-01-21T19:35:44.324 INFO:teuthology.orchestra.run.smithi169.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-21T19:35:44.325 INFO:teuthology.orchestra.run.smithi169.stdout:Group : System Environment/Base 2024-01-21T19:35:44.325 INFO:teuthology.orchestra.run.smithi169.stdout:Size : 9265586 2024-01-21T19:35:44.325 INFO:teuthology.orchestra.run.smithi169.stdout:License : GPLv3+ 2024-01-21T19:35:44.325 INFO:teuthology.orchestra.run.smithi169.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-21T19:35:44.325 INFO:teuthology.orchestra.run.smithi169.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-21T19:35:44.325 INFO:teuthology.orchestra.run.smithi169.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-21T19:35:44.325 INFO:teuthology.orchestra.run.smithi169.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-21T19:35:44.325 INFO:teuthology.orchestra.run.smithi169.stdout:Relocations : (not relocatable) 2024-01-21T19:35:44.325 INFO:teuthology.orchestra.run.smithi169.stdout:Packager : CentOS BuildSystem 2024-01-21T19:35:44.325 INFO:teuthology.orchestra.run.smithi169.stdout:Vendor : CentOS 2024-01-21T19:35:44.325 INFO:teuthology.orchestra.run.smithi169.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-21T19:35:44.325 INFO:teuthology.orchestra.run.smithi169.stdout:Summary : Support tools for GRUB. 2024-01-21T19:35:44.326 INFO:teuthology.orchestra.run.smithi169.stdout:Description : 2024-01-21T19:35:44.326 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:35:44.326 INFO:teuthology.orchestra.run.smithi169.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-21T19:35:44.326 INFO:teuthology.orchestra.run.smithi169.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-21T19:35:44.326 INFO:teuthology.orchestra.run.smithi169.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-21T19:35:44.326 INFO:teuthology.orchestra.run.smithi169.stdout:hardware devices. 2024-01-21T19:35:44.326 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:35:44.326 INFO:teuthology.orchestra.run.smithi169.stdout:This subpackage provides tools for support of all platforms. 2024-01-21T19:35:44.327 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-21T19:35:44.327 INFO:teuthology.task.kernel:Updating grub on smithi169 to boot 4.18.0-535.el8.x86_64 2024-01-21T19:35:44.327 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-21T19:35:44.514 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:35:44.514 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-4.18.0-535.el8.x86_64 1/3 2024-01-21T19:35:44.514 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-core-4.18.0-535.el8.x86_64 2/3 2024-01-21T19:35:44.979 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:35:44.980 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:35:44.980 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-01-21T19:35:44.980 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-4.18.0-535.el8.x86_64 kernel-core-4.18.0-535.el8.x86_64 2024-01-21T19:35:44.980 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-4.18.0-535.el8.x86_64 2024-01-21T19:35:44.980 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:35:44.980 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:35:45.176 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T19:35:45.391 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T19:35:45.392 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T19:35:45.392 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-535.el8.x86_64 2024-01-21T19:35:45.392 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-21T19:35:45.392 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-21T19:35:45.392 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-01-21T19:35:45.392 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi173.front.sepia.ceph.com, path=None, version=distro) 2024-01-21T19:35:45.392 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T19:35:45.603 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T19:35:45.603 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T19:35:45.603 DEBUG:teuthology.orchestra.run.smithi173:> sudo rpm -qi grub2-tools 2024-01-21T19:35:45.637 INFO:teuthology.orchestra.run.smithi173.stdout:Name : grub2-tools 2024-01-21T19:35:45.637 INFO:teuthology.orchestra.run.smithi173.stdout:Epoch : 1 2024-01-21T19:35:45.637 INFO:teuthology.orchestra.run.smithi173.stdout:Version : 2.02 2024-01-21T19:35:45.637 INFO:teuthology.orchestra.run.smithi173.stdout:Release : 129.el8 2024-01-21T19:35:45.637 INFO:teuthology.orchestra.run.smithi173.stdout:Architecture: x86_64 2024-01-21T19:35:45.637 INFO:teuthology.orchestra.run.smithi173.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-21T19:35:45.637 INFO:teuthology.orchestra.run.smithi173.stdout:Group : System Environment/Base 2024-01-21T19:35:45.637 INFO:teuthology.orchestra.run.smithi173.stdout:Size : 9265586 2024-01-21T19:35:45.637 INFO:teuthology.orchestra.run.smithi173.stdout:License : GPLv3+ 2024-01-21T19:35:45.637 INFO:teuthology.orchestra.run.smithi173.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-21T19:35:45.637 INFO:teuthology.orchestra.run.smithi173.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-21T19:35:45.637 INFO:teuthology.orchestra.run.smithi173.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-21T19:35:45.637 INFO:teuthology.orchestra.run.smithi173.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-21T19:35:45.637 INFO:teuthology.orchestra.run.smithi173.stdout:Relocations : (not relocatable) 2024-01-21T19:35:45.638 INFO:teuthology.orchestra.run.smithi173.stdout:Packager : CentOS BuildSystem 2024-01-21T19:35:45.638 INFO:teuthology.orchestra.run.smithi173.stdout:Vendor : CentOS 2024-01-21T19:35:45.638 INFO:teuthology.orchestra.run.smithi173.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-21T19:35:45.638 INFO:teuthology.orchestra.run.smithi173.stdout:Summary : Support tools for GRUB. 2024-01-21T19:35:45.638 INFO:teuthology.orchestra.run.smithi173.stdout:Description : 2024-01-21T19:35:45.638 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:35:45.638 INFO:teuthology.orchestra.run.smithi173.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-21T19:35:45.638 INFO:teuthology.orchestra.run.smithi173.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-21T19:35:45.638 INFO:teuthology.orchestra.run.smithi173.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-21T19:35:45.638 INFO:teuthology.orchestra.run.smithi173.stdout:hardware devices. 2024-01-21T19:35:45.638 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:35:45.638 INFO:teuthology.orchestra.run.smithi173.stdout:This subpackage provides tools for support of all platforms. 2024-01-21T19:35:45.639 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-21T19:35:45.639 INFO:teuthology.task.kernel:Updating grub on smithi173 to boot 4.18.0-535.el8.x86_64 2024-01-21T19:35:45.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-21T19:35:46.043 INFO:teuthology.orchestra.run.smithi169.stderr:Generating grub configuration file ... 2024-01-21T19:35:47.462 INFO:teuthology.orchestra.run.smithi173.stderr:Generating grub configuration file ... 2024-01-21T19:35:48.137 INFO:teuthology.orchestra.run.smithi169.stderr:done 2024-01-21T19:35:48.139 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2024-01-21T19:35:48.171 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.ar3v1HhkXL 2024-01-21T19:35:48.171 DEBUG:teuthology.orchestra.run.smithi169:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ar3v1HhkXL 2024-01-21T19:35:48.244 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.ar3v1HhkXL 2024-01-21T19:35:48.405 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.ar3v1HhkXL is 6KB 2024-01-21T19:35:48.454 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.ar3v1HhkXL 2024-01-21T19:35:48.471 DEBUG:teuthology.orchestra.run.smithi169:> sudo /bin/ls /boot/loader/entries || true 2024-01-21T19:35:48.536 INFO:teuthology.orchestra.run.smithi169.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-21T19:35:48.536 INFO:teuthology.orchestra.run.smithi169.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-21T19:35:48.537 INFO:teuthology.orchestra.run.smithi169.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-21T19:35:48.537 INFO:teuthology.orchestra.run.smithi169.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64.conf 2024-01-21T19:35:48.538 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64 2024-01-21T19:35:48.715 DEBUG:teuthology.orchestra.run.smithi169:> sudo shutdown -r now 2024-01-21T19:35:49.622 INFO:teuthology.orchestra.run.smithi173.stderr:done 2024-01-21T19:35:49.623 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-01-21T19:35:49.655 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.4MsbOjvlMZ 2024-01-21T19:35:49.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4MsbOjvlMZ 2024-01-21T19:35:49.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.4MsbOjvlMZ 2024-01-21T19:35:49.880 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.4MsbOjvlMZ is 6KB 2024-01-21T19:35:49.928 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.4MsbOjvlMZ 2024-01-21T19:35:49.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo /bin/ls /boot/loader/entries || true 2024-01-21T19:35:50.018 INFO:teuthology.orchestra.run.smithi173.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-21T19:35:50.018 INFO:teuthology.orchestra.run.smithi173.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-21T19:35:50.018 INFO:teuthology.orchestra.run.smithi173.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-21T19:35:50.018 INFO:teuthology.orchestra.run.smithi173.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64.conf 2024-01-21T19:35:50.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64 2024-01-21T19:35:50.184 DEBUG:teuthology.orchestra.run.smithi173:> sudo shutdown -r now 2024-01-21T19:36:18.744 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-01-21T19:36:18.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-01-21T19:36:18.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:36:20.188 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-01-21T19:36:20.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-01-21T19:36:20.189 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:36:37.184 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-01-21T19:36:38.716 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-01-21T19:36:46.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-01-21T19:36:46.193 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:36:47.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-01-21T19:36:47.719 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:36:49.248 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-01-21T19:36:50.780 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-01-21T19:37:01.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-01-21T19:37:01.258 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:37:02.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-01-21T19:37:02.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:37:34.869 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-01-21T19:37:36.235 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-01-21T19:37:36.236 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-535.el8.x86_64"... 2024-01-21T19:37:36.236 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-01-21T19:37:36.580 INFO:teuthology.orchestra.run.smithi173.stdout:4.18.0-535.el8.x86_64 2024-01-21T19:37:36.580 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-535.el8.x86_64 vs 4.18.0-535.el8.x86_64 2024-01-21T19:37:36.580 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-21T19:37:36.580 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-21T19:37:37.581 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-01-21T19:37:37.581 DEBUG:teuthology.orchestra.run.smithi173:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-21T19:37:37.663 INFO:teuthology.orchestra.run.smithi173.stdout:ttyS1 2024-01-21T19:37:37.677 DEBUG:teuthology.parallel:result is None 2024-01-21T19:38:01.260 DEBUG:teuthology.orchestra.remote:timed out 2024-01-21T19:38:16.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-01-21T19:38:16.262 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:38:16.564 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-01-21T19:38:16.860 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-01-21T19:38:16.860 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-535.el8.x86_64"... 2024-01-21T19:38:16.861 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-01-21T19:38:16.879 INFO:teuthology.orchestra.run.smithi169.stdout:4.18.0-535.el8.x86_64 2024-01-21T19:38:16.879 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-535.el8.x86_64 vs 4.18.0-535.el8.x86_64 2024-01-21T19:38:16.879 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-21T19:38:16.879 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-21T19:38:17.880 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-01-21T19:38:17.880 DEBUG:teuthology.orchestra.run.smithi169:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-21T19:38:17.966 INFO:teuthology.orchestra.run.smithi169.stdout:ttyS1 2024-01-21T19:38:17.980 DEBUG:teuthology.parallel:result is None 2024-01-21T19:38:17.980 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-21T19:38:17.991 INFO:teuthology.task.internal:Creating test directory... 2024-01-21T19:38:17.992 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-21T19:38:17.994 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-21T19:38:18.017 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-21T19:38:18.023 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-21T19:38:18.032 INFO:teuthology.task.internal:Creating archive directory... 2024-01-21T19:38:18.033 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-21T19:38:18.054 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-21T19:38:18.107 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-21T19:38:18.113 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-21T19:38:18.113 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-21T19:38:18.115 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-21T19:38:18.148 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T19:38:18.153 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T19:38:18.159 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T19:38:18.164 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T19:38:18.166 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-21T19:38:18.173 INFO:teuthology.task.internal:Configuring sudo... 2024-01-21T19:38:18.173 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-21T19:38:18.204 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-21T19:38:18.235 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-21T19:38:18.244 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-21T19:38:18.244 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-21T19:38:18.274 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-21T19:38:18.295 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T19:38:18.376 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T19:38:18.432 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T19:38:18.516 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T19:38:18.559 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-01-21T19:38:18.559 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-21T19:38:18.626 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T19:38:18.673 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T19:38:18.729 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T19:38:18.817 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T19:38:18.862 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-01-21T19:38:18.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-21T19:38:18.931 DEBUG:teuthology.orchestra.run.smithi169:> sudo service rsyslog restart 2024-01-21T19:38:18.934 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-01-21T19:38:18.989 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T19:38:19.049 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T19:38:19.431 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-21T19:38:19.438 INFO:teuthology.task.internal:Starting timer... 2024-01-21T19:38:19.439 INFO:teuthology.run_tasks:Running task pcp... 2024-01-21T19:38:19.447 INFO:teuthology.run_tasks:Running task selinux... 2024-01-21T19:38:19.455 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-21T19:38:19.455 DEBUG:teuthology.orchestra.run.smithi169:> sudo service auditd rotate 2024-01-21T19:38:19.559 INFO:teuthology.orchestra.run.smithi169.stdout:Rotating logs: 2024-01-21T19:38:19.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-01-21T19:38:19.654 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-01-21T19:38:19.655 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-21T19:38:19.656 DEBUG:teuthology.orchestra.run.smithi169:> /usr/sbin/getenforce 2024-01-21T19:38:19.698 INFO:teuthology.orchestra.run.smithi169.stdout:Permissive 2024-01-21T19:38:19.698 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-01-21T19:38:19.724 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-01-21T19:38:19.725 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi169.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive'} 2024-01-21T19:38:19.725 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T19:38:19.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T19:38:19.765 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T19:38:19.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T19:38:19.793 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-21T19:38:19.793 DEBUG:teuthology.orchestra.run.smithi169:> sudo /usr/sbin/setenforce permissive 2024-01-21T19:38:19.833 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-01-21T19:38:19.859 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-21T19:38:19.868 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-21T19:38:19.884 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-21T19:38:19.885 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi169.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-21T19:43:56.824 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi169.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2024-01-21T19:43:56.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-01-21T19:43:56.827 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:43:56.899 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-01-21T19:43:56.966 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-01-21T19:43:56.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-01-21T19:43:56.967 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:43:57.039 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-01-21T19:43:57.108 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-01-21T19:43:57.108 INFO:teuthology.run_tasks:Running task clock... 2024-01-21T19:43:57.119 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-21T19:43:57.119 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-21T19:43:57.119 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T19:43:57.121 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-21T19:43:57.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T19:43:57.151 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-21T19:43:57.164 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-21T19:43:57.181 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-21T19:43:57.196 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-21T19:43:57.230 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: ntpd: command not found 2024-01-21T19:43:57.241 INFO:teuthology.orchestra.run.smithi169.stdout:506 Cannot talk to daemon 2024-01-21T19:43:57.254 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-21T19:43:57.268 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-21T19:43:57.281 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-01-21T19:43:57.292 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-01-21T19:43:57.308 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-21T19:43:57.320 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-21T19:43:57.333 INFO:teuthology.orchestra.run.smithi169.stderr:bash: ntpq: command not found 2024-01-21T19:43:57.337 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T19:43:57.337 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-01-21T19:43:57.337 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:43:57.337 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:43:57.337 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:43:57.337 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:43:57.384 INFO:teuthology.orchestra.run.smithi173.stderr:bash: ntpq: command not found 2024-01-21T19:43:57.388 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T19:43:57.388 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-01-21T19:43:57.388 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:43:57.388 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:43:57.388 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:43:57.388 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:43:57.389 INFO:teuthology.run_tasks:Running task pexec... 2024-01-21T19:43:57.398 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-21T19:43:57.398 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi169.front.sepia.ceph.com 2024-01-21T19:43:57.398 DEBUG:teuthology.orchestra.run.smithi169:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T19:43:57.399 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi173.front.sepia.ceph.com 2024-01-21T19:43:57.399 DEBUG:teuthology.orchestra.run.smithi173:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T19:43:58.154 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:22 ago on Sun 21 Jan 2024 07:42:35 PM UTC. 2024-01-21T19:43:58.189 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:22 ago on Sun 21 Jan 2024 07:42:36 PM UTC. 2024-01-21T19:44:00.095 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:44:00.096 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:44:00.096 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-01-21T19:44:00.096 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:44:00.096 INFO:teuthology.orchestra.run.smithi169.stdout:Resetting modules: 2024-01-21T19:44:00.096 INFO:teuthology.orchestra.run.smithi169.stdout: container-tools 2024-01-21T19:44:00.096 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:44:00.096 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-21T19:44:00.096 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:44:00.096 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:44:00.133 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:44:00.134 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:44:00.134 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-01-21T19:44:00.134 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:44:00.134 INFO:teuthology.orchestra.run.smithi173.stdout:Resetting modules: 2024-01-21T19:44:00.134 INFO:teuthology.orchestra.run.smithi173.stdout: container-tools 2024-01-21T19:44:00.135 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:44:00.135 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-01-21T19:44:00.135 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:44:00.135 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:44:00.347 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:44:00.380 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:44:01.169 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:26 ago on Sun 21 Jan 2024 07:42:35 PM UTC. 2024-01-21T19:44:01.204 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:25 ago on Sun 21 Jan 2024 07:42:36 PM UTC. 2024-01-21T19:44:03.091 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:44:03.125 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:44:03.125 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-01-21T19:44:03.125 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:44:03.125 INFO:teuthology.orchestra.run.smithi169.stdout:Installing group/module packages: 2024-01-21T19:44:03.125 INFO:teuthology.orchestra.run.smithi169.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-21T19:44:03.125 INFO:teuthology.orchestra.run.smithi169.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-21T19:44:03.125 INFO:teuthology.orchestra.run.smithi169.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-21T19:44:03.125 INFO:teuthology.orchestra.run.smithi169.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-21T19:44:03.125 INFO:teuthology.orchestra.run.smithi169.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-21T19:44:03.125 INFO:teuthology.orchestra.run.smithi169.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-21T19:44:03.125 INFO:teuthology.orchestra.run.smithi169.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-21T19:44:03.125 INFO:teuthology.orchestra.run.smithi169.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-21T19:44:03.126 INFO:teuthology.orchestra.run.smithi169.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-21T19:44:03.126 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-01-21T19:44:03.126 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-21T19:44:03.126 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-21T19:44:03.126 INFO:teuthology.orchestra.run.smithi169.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-21T19:44:03.126 INFO:teuthology.orchestra.run.smithi169.stdout:Installing weak dependencies: 2024-01-21T19:44:03.126 INFO:teuthology.orchestra.run.smithi169.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-21T19:44:03.126 INFO:teuthology.orchestra.run.smithi169.stdout:Installing module profiles: 2024-01-21T19:44:03.126 INFO:teuthology.orchestra.run.smithi169.stdout: container-tools/common 2024-01-21T19:44:03.126 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:44:03.126 INFO:teuthology.orchestra.run.smithi169.stdout:Enabling module streams: 2024-01-21T19:44:03.126 INFO:teuthology.orchestra.run.smithi169.stdout: container-tools rhel8 2024-01-21T19:44:03.126 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:44:03.126 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-21T19:44:03.126 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:44:03.126 INFO:teuthology.orchestra.run.smithi169.stdout:Install 13 Packages 2024-01-21T19:44:03.127 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:44:03.127 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 27 M 2024-01-21T19:44:03.127 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 89 M 2024-01-21T19:44:03.127 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-21T19:44:03.145 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:44:03.180 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:44:03.180 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-01-21T19:44:03.180 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:44:03.180 INFO:teuthology.orchestra.run.smithi173.stdout:Installing group/module packages: 2024-01-21T19:44:03.180 INFO:teuthology.orchestra.run.smithi173.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-21T19:44:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-21T19:44:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-21T19:44:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-21T19:44:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-21T19:44:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-21T19:44:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-21T19:44:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-21T19:44:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-21T19:44:03.181 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-01-21T19:44:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-21T19:44:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-21T19:44:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-21T19:44:03.181 INFO:teuthology.orchestra.run.smithi173.stdout:Installing weak dependencies: 2024-01-21T19:44:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-21T19:44:03.181 INFO:teuthology.orchestra.run.smithi173.stdout:Installing module profiles: 2024-01-21T19:44:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: container-tools/common 2024-01-21T19:44:03.182 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:44:03.182 INFO:teuthology.orchestra.run.smithi173.stdout:Enabling module streams: 2024-01-21T19:44:03.182 INFO:teuthology.orchestra.run.smithi173.stdout: container-tools rhel8 2024-01-21T19:44:03.182 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:44:03.182 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-01-21T19:44:03.182 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:44:03.182 INFO:teuthology.orchestra.run.smithi173.stdout:Install 13 Packages 2024-01-21T19:44:03.182 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:44:03.183 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 27 M 2024-01-21T19:44:03.183 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 89 M 2024-01-21T19:44:03.183 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-01-21T19:44:04.102 INFO:teuthology.orchestra.run.smithi173.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.6 MB/s | 750 kB 00:00 2024-01-21T19:44:04.136 INFO:teuthology.orchestra.run.smithi173.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 3.2 MB/s | 1.0 MB 00:00 2024-01-21T19:44:04.170 INFO:teuthology.orchestra.run.smithi173.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 567 kB/s | 38 kB 00:00 2024-01-21T19:44:04.212 INFO:teuthology.orchestra.run.smithi173.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 3.1 MB/s | 241 kB 00:00 2024-01-21T19:44:04.312 INFO:teuthology.orchestra.run.smithi173.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.7 MB/s | 178 kB 00:00 2024-01-21T19:44:04.421 INFO:teuthology.orchestra.run.smithi173.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 229 kB/s | 25 kB 00:00 2024-01-21T19:44:04.521 INFO:teuthology.orchestra.run.smithi173.stdout:(7/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 14 MB/s | 9.6 MB 00:00 2024-01-21T19:44:04.546 INFO:teuthology.orchestra.run.smithi173.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 749 kB/s | 94 kB 00:00 2024-01-21T19:44:04.613 INFO:teuthology.orchestra.run.smithi173.stdout:(9/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 8.8 MB/s | 3.9 MB 00:00 2024-01-21T19:44:04.680 INFO:teuthology.orchestra.run.smithi173.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 788 kB/s | 53 kB 00:00 2024-01-21T19:44:04.739 INFO:teuthology.orchestra.run.smithi173.stdout:(11/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 13 MB/s | 2.5 MB 00:00 2024-01-21T19:44:04.764 INFO:teuthology.orchestra.run.smithi173.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 491 kB/s | 41 kB 00:00 2024-01-21T19:44:04.907 INFO:teuthology.orchestra.run.smithi173.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 21 MB/s | 8.1 MB 00:00 2024-01-21T19:44:04.907 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:44:04.907 INFO:teuthology.orchestra.run.smithi173.stdout:Total 15 MB/s | 27 MB 00:01 2024-01-21T19:44:05.173 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-01-21T19:44:05.194 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-01-21T19:44:05.194 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-01-21T19:44:05.198 INFO:teuthology.orchestra.run.smithi169.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.5 MB/s | 750 kB 00:00 2024-01-21T19:44:05.231 INFO:teuthology.orchestra.run.smithi169.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 3.1 MB/s | 1.0 MB 00:00 2024-01-21T19:44:05.265 INFO:teuthology.orchestra.run.smithi169.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 568 kB/s | 38 kB 00:00 2024-01-21T19:44:05.307 INFO:teuthology.orchestra.run.smithi169.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 3.1 MB/s | 241 kB 00:00 2024-01-21T19:44:05.408 INFO:teuthology.orchestra.run.smithi169.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.7 MB/s | 178 kB 00:00 2024-01-21T19:44:05.452 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-01-21T19:44:05.455 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-01-21T19:44:05.483 INFO:teuthology.orchestra.run.smithi169.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 330 kB/s | 25 kB 00:00 2024-01-21T19:44:05.600 INFO:teuthology.orchestra.run.smithi169.stdout:(7/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 14 MB/s | 9.6 MB 00:00 2024-01-21T19:44:05.625 INFO:teuthology.orchestra.run.smithi169.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 661 kB/s | 94 kB 00:00 2024-01-21T19:44:05.692 INFO:teuthology.orchestra.run.smithi169.stdout:(9/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 9.2 MB/s | 3.9 MB 00:00 2024-01-21T19:44:05.751 INFO:teuthology.orchestra.run.smithi169.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 904 kB/s | 53 kB 00:00 2024-01-21T19:44:05.809 INFO:teuthology.orchestra.run.smithi169.stdout:(11/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 14 MB/s | 2.5 MB 00:00 2024-01-21T19:44:05.835 INFO:teuthology.orchestra.run.smithi169.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 490 kB/s | 41 kB 00:00 2024-01-21T19:44:05.944 INFO:teuthology.orchestra.run.smithi169.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 24 MB/s | 8.1 MB 00:00 2024-01-21T19:44:05.944 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:44:05.944 INFO:teuthology.orchestra.run.smithi169.stdout:Total 9.4 MB/s | 27 MB 00:02 2024-01-21T19:44:06.156 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-01-21T19:44:06.209 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-21T19:44:06.231 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-21T19:44:06.231 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-21T19:44:06.388 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-21T19:44:06.488 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-21T19:44:06.491 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-21T19:44:06.542 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-21T19:44:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-21T19:44:07.010 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T19:44:07.170 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-21T19:44:07.402 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-21T19:44:07.565 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-21T19:44:07.581 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-21T19:44:07.775 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-21T19:44:08.024 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T19:44:08.566 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-21T19:44:08.584 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T19:44:08.829 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-21T19:44:09.022 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-21T19:44:09.280 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-21T19:44:09.533 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T19:44:09.655 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-21T19:44:09.842 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-21T19:44:10.065 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-21T19:44:10.369 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-21T19:44:10.526 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T19:44:10.778 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-21T19:44:10.797 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-21T19:44:10.966 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T19:44:11.409 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T19:44:11.409 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-21T19:44:11.410 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-21T19:44:11.410 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-21T19:44:11.410 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T19:44:11.410 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-21T19:44:11.410 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T19:44:11.410 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-21T19:44:11.410 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-21T19:44:11.410 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-21T19:44:11.410 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-21T19:44:11.410 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T19:44:11.410 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-21T19:44:11.691 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T19:44:11.907 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-21T19:44:11.963 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-21T19:44:11.963 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:44:11.963 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-01-21T19:44:11.963 INFO:teuthology.orchestra.run.smithi173.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:44:11.963 INFO:teuthology.orchestra.run.smithi173.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:44:11.964 INFO:teuthology.orchestra.run.smithi173.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-21T19:44:11.964 INFO:teuthology.orchestra.run.smithi173.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-21T19:44:11.964 INFO:teuthology.orchestra.run.smithi173.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:44:11.964 INFO:teuthology.orchestra.run.smithi173.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:44:11.964 INFO:teuthology.orchestra.run.smithi173.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-21T19:44:11.964 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-21T19:44:11.964 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-21T19:44:11.964 INFO:teuthology.orchestra.run.smithi173.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:44:11.964 INFO:teuthology.orchestra.run.smithi173.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:44:11.964 INFO:teuthology.orchestra.run.smithi173.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-21T19:44:11.964 INFO:teuthology.orchestra.run.smithi173.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-21T19:44:11.964 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:44:11.964 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:44:12.121 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T19:44:12.236 DEBUG:teuthology.parallel:result is None 2024-01-21T19:44:12.541 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T19:44:12.541 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-21T19:44:12.542 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-21T19:44:12.542 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-21T19:44:12.542 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T19:44:12.542 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-21T19:44:12.542 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T19:44:12.542 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-21T19:44:12.542 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-21T19:44:12.542 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-21T19:44:12.542 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-21T19:44:12.542 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T19:44:12.542 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-21T19:44:13.093 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-21T19:44:13.094 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:44:13.094 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-21T19:44:13.094 INFO:teuthology.orchestra.run.smithi169.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:44:13.094 INFO:teuthology.orchestra.run.smithi169.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:44:13.094 INFO:teuthology.orchestra.run.smithi169.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-21T19:44:13.094 INFO:teuthology.orchestra.run.smithi169.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-21T19:44:13.094 INFO:teuthology.orchestra.run.smithi169.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:44:13.094 INFO:teuthology.orchestra.run.smithi169.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:44:13.094 INFO:teuthology.orchestra.run.smithi169.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-21T19:44:13.094 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-21T19:44:13.094 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-21T19:44:13.094 INFO:teuthology.orchestra.run.smithi169.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:44:13.094 INFO:teuthology.orchestra.run.smithi169.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:44:13.094 INFO:teuthology.orchestra.run.smithi169.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-21T19:44:13.095 INFO:teuthology.orchestra.run.smithi169.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-21T19:44:13.095 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:44:13.095 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:44:13.387 DEBUG:teuthology.parallel:result is None 2024-01-21T19:44:13.387 INFO:teuthology.run_tasks:Running task pexec... 2024-01-21T19:44:13.395 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-21T19:44:13.396 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi169.front.sepia.ceph.com 2024-01-21T19:44:13.396 DEBUG:teuthology.orchestra.run.smithi169:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T19:44:13.396 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi173.front.sepia.ceph.com 2024-01-21T19:44:13.397 DEBUG:teuthology.orchestra.run.smithi173:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T19:44:13.564 DEBUG:teuthology.parallel:result is None 2024-01-21T19:44:13.591 DEBUG:teuthology.parallel:result is None 2024-01-21T19:44:13.591 INFO:teuthology.run_tasks:Running task install... 2024-01-21T19:44:13.600 DEBUG:teuthology.task.install:project ceph 2024-01-21T19:44:13.600 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec'}} 2024-01-21T19:44:13.600 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec'} 2024-01-21T19:44:13.600 INFO:teuthology.task.install:Using flavor: default 2024-01-21T19:44:13.608 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-21T19:44:13.608 INFO:teuthology.task.install:extra packages: [] 2024-01-21T19:44:13.608 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-21T19:44:13.608 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T19:44:13.608 INFO:teuthology.packaging:ref: None 2024-01-21T19:44:13.609 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T19:44:13.609 INFO:teuthology.packaging:branch: None 2024-01-21T19:44:13.609 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T19:44:14.156 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T19:44:14.156 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T19:44:14.159 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-21T19:44:14.159 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T19:44:14.159 INFO:teuthology.packaging:ref: None 2024-01-21T19:44:14.159 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T19:44:14.159 INFO:teuthology.packaging:branch: None 2024-01-21T19:44:14.159 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T19:44:14.159 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T19:44:14.308 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-21T19:44:14.308 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-21T19:44:14.338 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-21T19:44:14.339 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-01-21T19:44:14.339 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-21T19:44:14.372 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-21T19:44:14.373 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T19:44:14.373 INFO:teuthology.packaging:ref: None 2024-01-21T19:44:14.373 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T19:44:14.373 INFO:teuthology.packaging:branch: None 2024-01-21T19:44:14.373 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T19:44:14.373 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-21T19:44:14.376 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-21T19:44:14.376 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-21T19:44:14.399 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-21T19:44:14.400 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-01-21T19:44:14.400 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-21T19:44:14.427 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-21T19:44:14.427 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T19:44:14.428 INFO:teuthology.packaging:ref: None 2024-01-21T19:44:14.428 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T19:44:14.428 INFO:teuthology.packaging:branch: None 2024-01-21T19:44:14.428 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T19:44:14.428 DEBUG:teuthology.orchestra.run.smithi169:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-21T19:44:14.444 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-21T19:44:14.495 DEBUG:teuthology.orchestra.run.smithi169:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-21T19:44:14.524 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-21T19:44:14.573 DEBUG:teuthology.orchestra.run.smithi169:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-21T19:44:14.593 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2024-01-21T19:44:14.594 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2024-01-21T19:44:14.639 INFO:teuthology.orchestra.run.smithi169.stdout:check_obsoletes = 1 2024-01-21T19:44:14.640 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum clean all 2024-01-21T19:44:14.861 INFO:teuthology.orchestra.run.smithi173.stdout:76 files removed 2024-01-21T19:44:14.889 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw 2024-01-21T19:44:14.934 INFO:teuthology.orchestra.run.smithi169.stdout:76 files removed 2024-01-21T19:44:14.968 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-radosgw 2024-01-21T19:44:15.375 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 971 kB/s | 74 kB 00:00 2024-01-21T19:44:15.462 INFO:teuthology.orchestra.run.smithi169.stdout:ceph packages for x86_64 890 kB/s | 74 kB 00:00 2024-01-21T19:44:15.484 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 214 kB/s | 15 kB 00:00 2024-01-21T19:44:15.579 INFO:teuthology.orchestra.run.smithi169.stdout:ceph noarch packages 192 kB/s | 15 kB 00:00 2024-01-21T19:44:15.585 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2024-01-21T19:44:15.696 INFO:teuthology.orchestra.run.smithi169.stdout:ceph source packages 20 kB/s | 1.7 kB 00:00 2024-01-21T19:44:16.836 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - AppStream 22 MB/s | 26 MB 00:01 2024-01-21T19:44:16.972 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 8 - AppStream 21 MB/s | 26 MB 00:01 2024-01-21T19:44:22.679 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - BaseOS 8.4 MB/s | 10 MB 00:01 2024-01-21T19:44:22.883 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 8 - BaseOS 8.3 MB/s | 10 MB 00:01 2024-01-21T19:44:24.679 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - Extras 44 kB/s | 18 kB 00:00 2024-01-21T19:44:24.896 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 8 - Extras 42 kB/s | 18 kB 00:00 2024-01-21T19:44:25.146 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2024-01-21T19:44:25.357 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 6.9 kB 00:00 2024-01-21T19:44:26.606 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - PowerTools 3.5 MB/s | 4.7 MB 00:01 2024-01-21T19:44:26.709 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 8 - PowerTools 3.8 MB/s | 4.7 MB 00:01 2024-01-21T19:44:27.608 INFO:teuthology.orchestra.run.smithi173.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-01-21T19:44:27.735 INFO:teuthology.orchestra.run.smithi169.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-01-21T19:44:29.353 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-01-21T19:44:29.570 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 8.6 MB/s | 16 MB 00:01 2024-01-21T19:44:33.177 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 466 kB/s | 24 kB 00:00 2024-01-21T19:44:33.425 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 401 kB/s | 24 kB 00:00 2024-01-21T19:44:36.648 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:44:36.651 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:44:36.651 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-01-21T19:44:36.651 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:44:36.651 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-21T19:44:36.652 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-21T19:44:36.652 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrading: 2024-01-21T19:44:36.652 INFO:teuthology.orchestra.run.smithi169.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T19:44:36.652 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-21T19:44:36.652 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-01-21T19:44:36.652 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-21T19:44:36.652 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-21T19:44:36.652 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-21T19:44:36.652 INFO:teuthology.orchestra.run.smithi169.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-21T19:44:36.652 INFO:teuthology.orchestra.run.smithi169.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-21T19:44:36.652 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-21T19:44:36.652 INFO:teuthology.orchestra.run.smithi169.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-21T19:44:36.652 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-21T19:44:36.652 INFO:teuthology.orchestra.run.smithi169.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-21T19:44:36.653 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-21T19:44:36.653 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-21T19:44:36.653 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-21T19:44:36.653 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-21T19:44:36.653 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-21T19:44:36.653 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-21T19:44:36.653 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-21T19:44:36.653 INFO:teuthology.orchestra.run.smithi169.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-21T19:44:36.653 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-21T19:44:36.653 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-21T19:44:36.653 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-21T19:44:36.653 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-21T19:44:36.654 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-21T19:44:36.654 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-21T19:44:36.654 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-21T19:44:36.654 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:44:36.654 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-21T19:44:36.654 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:44:36.654 INFO:teuthology.orchestra.run.smithi169.stdout:Install 25 Packages 2024-01-21T19:44:36.654 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrade 2 Packages 2024-01-21T19:44:36.654 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:44:36.655 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 59 M 2024-01-21T19:44:36.655 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-21T19:44:36.887 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:44:36.890 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:44:36.890 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-01-21T19:44:36.890 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:44:36.890 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-01-21T19:44:36.890 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-21T19:44:36.890 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-01-21T19:44:36.890 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T19:44:36.890 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-21T19:44:36.890 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-01-21T19:44:36.891 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-21T19:44:36.891 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-21T19:44:36.891 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-21T19:44:36.891 INFO:teuthology.orchestra.run.smithi173.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-21T19:44:36.891 INFO:teuthology.orchestra.run.smithi173.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-21T19:44:36.891 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-21T19:44:36.891 INFO:teuthology.orchestra.run.smithi173.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-21T19:44:36.891 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-21T19:44:36.891 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-21T19:44:36.891 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-21T19:44:36.891 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-21T19:44:36.891 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-21T19:44:36.892 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-21T19:44:36.892 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-21T19:44:36.892 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-21T19:44:36.892 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-21T19:44:36.892 INFO:teuthology.orchestra.run.smithi173.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-21T19:44:36.892 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-21T19:44:36.892 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-21T19:44:36.892 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-21T19:44:36.892 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-21T19:44:36.892 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-21T19:44:36.892 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-21T19:44:36.892 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-21T19:44:36.892 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:44:36.892 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-01-21T19:44:36.892 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:44:36.893 INFO:teuthology.orchestra.run.smithi173.stdout:Install 25 Packages 2024-01-21T19:44:36.893 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 2 Packages 2024-01-21T19:44:36.893 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:44:36.893 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 59 M 2024-01-21T19:44:36.893 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-01-21T19:44:37.537 INFO:teuthology.orchestra.run.smithi169.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 44 MB/s | 6.0 MB 00:00 2024-01-21T19:44:37.562 INFO:teuthology.orchestra.run.smithi169.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-01-21T19:44:37.585 INFO:teuthology.orchestra.run.smithi173.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 35 MB/s | 6.0 MB 00:00 2024-01-21T19:44:37.604 INFO:teuthology.orchestra.run.smithi169.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2024-01-21T19:44:37.610 INFO:teuthology.orchestra.run.smithi173.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-01-21T19:44:37.638 INFO:teuthology.orchestra.run.smithi169.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-01-21T19:44:37.652 INFO:teuthology.orchestra.run.smithi173.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2024-01-21T19:44:37.720 INFO:teuthology.orchestra.run.smithi173.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 7.0 MB/s | 486 kB 00:00 2024-01-21T19:44:37.730 INFO:teuthology.orchestra.run.smithi169.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 44 MB/s | 4.1 MB 00:00 2024-01-21T19:44:37.780 INFO:teuthology.orchestra.run.smithi169.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 893 kB/s | 44 kB 00:00 2024-01-21T19:44:37.798 INFO:teuthology.orchestra.run.smithi169.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 4.7 MB/s | 80 kB 00:00 2024-01-21T19:44:37.812 INFO:teuthology.orchestra.run.smithi173.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 45 MB/s | 4.1 MB 00:00 2024-01-21T19:44:37.822 INFO:teuthology.orchestra.run.smithi169.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2024-01-21T19:44:37.837 INFO:teuthology.orchestra.run.smithi173.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.7 MB/s | 44 kB 00:00 2024-01-21T19:44:37.848 INFO:teuthology.orchestra.run.smithi169.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 15 MB/s | 387 kB 00:00 2024-01-21T19:44:37.862 INFO:teuthology.orchestra.run.smithi173.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-01-21T19:44:37.887 INFO:teuthology.orchestra.run.smithi173.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2024-01-21T19:44:37.921 INFO:teuthology.orchestra.run.smithi173.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-01-21T19:44:37.982 INFO:teuthology.orchestra.run.smithi169.stdout:(10/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 22 MB/s | 13 MB 00:00 2024-01-21T19:44:38.007 INFO:teuthology.orchestra.run.smithi169.stdout:(11/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 2.3 MB/s | 368 kB 00:00 2024-01-21T19:44:38.040 INFO:teuthology.orchestra.run.smithi169.stdout:(12/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 1.9 MB/s | 114 kB 00:00 2024-01-21T19:44:38.046 INFO:teuthology.orchestra.run.smithi173.stdout:(10/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 20 MB/s | 13 MB 00:00 2024-01-21T19:44:38.063 INFO:teuthology.orchestra.run.smithi173.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 6.9 MB/s | 114 kB 00:00 2024-01-21T19:44:38.088 INFO:teuthology.orchestra.run.smithi173.stdout:(12/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 2.1 MB/s | 368 kB 00:00 2024-01-21T19:44:38.328 INFO:teuthology.orchestra.run.smithi169.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 26 MB/s | 24 MB 00:00 2024-01-21T19:44:38.358 INFO:teuthology.orchestra.run.smithi169.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 414 kB/s | 145 kB 00:00 2024-01-21T19:44:38.372 INFO:teuthology.orchestra.run.smithi173.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 25 MB/s | 24 MB 00:00 2024-01-21T19:44:38.416 INFO:teuthology.orchestra.run.smithi169.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 754 kB/s | 44 kB 00:00 2024-01-21T19:44:38.443 INFO:teuthology.orchestra.run.smithi169.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.6 MB/s | 677 kB 00:00 2024-01-21T19:44:38.534 INFO:teuthology.orchestra.run.smithi169.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2024-01-21T19:44:38.567 INFO:teuthology.orchestra.run.smithi169.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 280 kB/s | 42 kB 00:00 2024-01-21T19:44:38.581 INFO:teuthology.orchestra.run.smithi173.stdout:(14/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2024-01-21T19:44:38.592 INFO:teuthology.orchestra.run.smithi169.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 312 kB/s | 47 kB 00:00 2024-01-21T19:44:38.606 INFO:teuthology.orchestra.run.smithi173.stdout:(15/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 267 kB/s | 145 kB 00:00 2024-01-21T19:44:38.665 INFO:teuthology.orchestra.run.smithi173.stdout:(16/27): python3-prettytable-0.7.2-14.el8.noarc 526 kB/s | 44 kB 00:00 2024-01-21T19:44:38.699 INFO:teuthology.orchestra.run.smithi173.stdout:(17/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.1 MB/s | 677 kB 00:00 2024-01-21T19:44:38.726 INFO:teuthology.orchestra.run.smithi169.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 417 kB/s | 80 kB 00:00 2024-01-21T19:44:38.751 INFO:teuthology.orchestra.run.smithi169.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 633 kB/s | 116 kB 00:00 2024-01-21T19:44:38.782 INFO:teuthology.orchestra.run.smithi173.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 240 kB/s | 42 kB 00:00 2024-01-21T19:44:38.785 INFO:teuthology.orchestra.run.smithi169.stdout:(22/27): liboath-2.6.2-3.el8.x86_64.rpm 2.0 MB/s | 59 kB 00:00 2024-01-21T19:44:38.810 INFO:teuthology.orchestra.run.smithi169.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.4 MB/s | 306 kB 00:00 2024-01-21T19:44:38.824 INFO:teuthology.orchestra.run.smithi173.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 296 kB/s | 47 kB 00:00 2024-01-21T19:44:38.835 INFO:teuthology.orchestra.run.smithi169.stdout:(24/27): libunwind-1.3.1-3.el8.x86_64.rpm 1.5 MB/s | 75 kB 00:00 2024-01-21T19:44:38.883 INFO:teuthology.orchestra.run.smithi173.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 436 kB/s | 80 kB 00:00 2024-01-21T19:44:38.919 INFO:teuthology.orchestra.run.smithi169.stdout:(25/27): librbd1-16.2.4-0.el8.x86_64.rpm 49 MB/s | 4.0 MB 00:00 2024-01-21T19:44:38.978 INFO:teuthology.orchestra.run.smithi169.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 22 MB/s | 3.7 MB 00:00 2024-01-21T19:44:39.003 INFO:teuthology.orchestra.run.smithi169.stdout:(27/27): leveldb-1.22-1.el8.x86_64.rpm 654 kB/s | 181 kB 00:00 2024-01-21T19:44:39.008 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:44:39.008 INFO:teuthology.orchestra.run.smithi169.stdout:Total 25 MB/s | 59 MB 00:02 2024-01-21T19:44:39.074 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-21T19:44:39.123 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-21T19:44:39.123 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-21T19:44:39.133 INFO:teuthology.orchestra.run.smithi173.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 332 kB/s | 116 kB 00:00 2024-01-21T19:44:39.217 INFO:teuthology.orchestra.run.smithi173.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 781 kB/s | 306 kB 00:00 2024-01-21T19:44:39.242 INFO:teuthology.orchestra.run.smithi173.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 542 kB/s | 59 kB 00:00 2024-01-21T19:44:39.267 INFO:teuthology.orchestra.run.smithi173.stdout:(24/27): leveldb-1.22-1.el8.x86_64.rpm 477 kB/s | 181 kB 00:00 2024-01-21T19:44:39.293 INFO:teuthology.orchestra.run.smithi173.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 993 kB/s | 75 kB 00:00 2024-01-21T19:44:39.376 INFO:teuthology.orchestra.run.smithi173.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 28 MB/s | 3.7 MB 00:00 2024-01-21T19:44:39.443 INFO:teuthology.orchestra.run.smithi173.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 23 MB/s | 4.0 MB 00:00 2024-01-21T19:44:39.448 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:44:39.448 INFO:teuthology.orchestra.run.smithi173.stdout:Total 23 MB/s | 59 MB 00:02 2024-01-21T19:44:39.484 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-21T19:44:39.487 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-21T19:44:39.519 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-01-21T19:44:39.569 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-01-21T19:44:39.569 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-01-21T19:44:39.928 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-01-21T19:44:39.931 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-01-21T19:44:40.590 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-21T19:44:40.805 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-21T19:44:41.018 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-21T19:44:41.077 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-01-21T19:44:41.192 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T19:44:41.617 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T19:44:42.143 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-21T19:44:42.340 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-21T19:44:42.459 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T19:44:42.545 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T19:44:42.558 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T19:44:42.791 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T19:44:42.951 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T19:44:42.975 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T19:44:43.028 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T19:44:43.186 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T19:44:43.236 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-21T19:44:43.288 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T19:44:43.575 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T19:44:43.745 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T19:44:43.770 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-21T19:44:43.791 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T19:44:43.909 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T19:44:43.983 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T19:44:44.030 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-21T19:44:44.164 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-21T19:44:44.325 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T19:44:44.407 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T19:44:44.571 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-21T19:44:44.579 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T19:44:44.720 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T19:44:44.775 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T19:44:44.799 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T19:44:44.967 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T19:44:45.024 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-21T19:44:45.043 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T19:44:45.203 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T19:44:45.212 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T19:44:45.273 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T19:44:45.473 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T19:44:45.646 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T19:44:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T19:44:45.792 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T19:44:45.887 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T19:44:45.894 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T19:44:45.959 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T19:44:46.054 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-21T19:44:46.111 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T19:44:46.235 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-21T19:44:46.366 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T19:44:46.466 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T19:44:46.524 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T19:44:46.659 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-21T19:44:46.702 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T19:44:46.816 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-21T19:44:46.839 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T19:44:46.982 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-21T19:44:47.031 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-21T19:44:47.217 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-21T19:44:47.247 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-21T19:44:47.494 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T19:44:47.577 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T19:44:47.795 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-21T19:44:47.977 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-21T19:44:48.159 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-21T19:44:48.403 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-21T19:44:49.849 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T19:44:50.067 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T19:44:50.782 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T19:44:50.902 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T19:44:51.067 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T19:44:51.136 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T19:44:51.169 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T19:44:51.316 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T19:44:51.844 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T19:44:52.175 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T19:44:52.265 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T19:44:52.402 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T19:45:10.478 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T19:45:10.478 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /sys 2024-01-21T19:45:10.479 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /proc 2024-01-21T19:45:10.479 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /dev 2024-01-21T19:45:10.479 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /run 2024-01-21T19:45:10.479 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /mnt 2024-01-21T19:45:10.479 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /var/tmp 2024-01-21T19:45:10.479 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /home 2024-01-21T19:45:10.479 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /tmp 2024-01-21T19:45:10.479 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /dev 2024-01-21T19:45:10.479 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:45:11.486 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T19:45:11.487 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2024-01-21T19:45:11.487 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2024-01-21T19:45:11.487 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /dev 2024-01-21T19:45:11.487 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /run 2024-01-21T19:45:11.487 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2024-01-21T19:45:11.487 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2024-01-21T19:45:11.487 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2024-01-21T19:45:11.487 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2024-01-21T19:45:11.487 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /dev 2024-01-21T19:45:11.487 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:45:11.797 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T19:45:11.874 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T19:45:11.874 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T19:45:12.089 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T19:45:12.090 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T19:45:12.771 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T19:45:12.771 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-21T19:45:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-21T19:45:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-21T19:45:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T19:45:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T19:45:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T19:45:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-21T19:45:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-21T19:45:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T19:45:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-21T19:45:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-21T19:45:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T19:45:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T19:45:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-21T19:45:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-21T19:45:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-21T19:45:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-21T19:45:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-21T19:45:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-21T19:45:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-21T19:45:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-21T19:45:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-21T19:45:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-21T19:45:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-21T19:45:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-21T19:45:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T19:45:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-21T19:45:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-21T19:45:12.796 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T19:45:12.891 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T19:45:12.891 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T19:45:13.109 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T19:45:13.109 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T19:45:13.699 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T19:45:13.699 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:45:13.699 INFO:teuthology.orchestra.run.smithi169.stdout:Upgraded: 2024-01-21T19:45:13.699 INFO:teuthology.orchestra.run.smithi169.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:13.699 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-21T19:45:13.699 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:13.700 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:13.700 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:13.700 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:13.700 INFO:teuthology.orchestra.run.smithi169.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-21T19:45:13.700 INFO:teuthology.orchestra.run.smithi169.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-21T19:45:13.700 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-21T19:45:13.700 INFO:teuthology.orchestra.run.smithi169.stdout: leveldb-1.22-1.el8.x86_64 2024-01-21T19:45:13.700 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:13.700 INFO:teuthology.orchestra.run.smithi169.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-21T19:45:13.700 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-21T19:45:13.700 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-21T19:45:13.700 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:13.700 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-21T19:45:13.700 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:13.701 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-21T19:45:13.701 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-21T19:45:13.701 INFO:teuthology.orchestra.run.smithi169.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-21T19:45:13.701 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:13.701 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:13.701 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:13.701 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-21T19:45:13.701 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:13.701 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:13.701 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:13.701 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:45:13.701 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:45:13.834 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T19:45:13.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-21T19:45:13.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-21T19:45:13.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-21T19:45:13.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T19:45:13.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T19:45:13.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T19:45:13.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-21T19:45:13.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-21T19:45:13.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T19:45:13.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-21T19:45:13.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-21T19:45:13.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T19:45:13.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T19:45:13.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-21T19:45:13.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-21T19:45:13.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-21T19:45:13.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-21T19:45:13.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-21T19:45:13.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-21T19:45:13.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-21T19:45:13.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-21T19:45:13.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-21T19:45:13.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-21T19:45:13.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-21T19:45:13.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-21T19:45:13.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T19:45:13.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-21T19:45:13.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-21T19:45:13.991 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-test 2024-01-21T19:45:14.718 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T19:45:14.718 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:45:14.718 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-01-21T19:45:14.718 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:14.718 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-01-21T19:45:14.718 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:14.718 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:14.718 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:14.718 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-21T19:45:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-21T19:45:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-21T19:45:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: leveldb-1.22-1.el8.x86_64 2024-01-21T19:45:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-21T19:45:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-21T19:45:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-21T19:45:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-21T19:45:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-21T19:45:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-21T19:45:14.720 INFO:teuthology.orchestra.run.smithi173.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-21T19:45:14.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:14.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:14.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:14.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-21T19:45:14.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:14.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:14.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:14.720 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:45:14.720 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:45:14.721 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:41 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:45:15.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-test 2024-01-21T19:45:15.735 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:42 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:45:16.180 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:45:16.181 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:45:16.181 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-01-21T19:45:16.181 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:45:16.181 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-21T19:45:16.181 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-21T19:45:16.181 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-01-21T19:45:16.181 INFO:teuthology.orchestra.run.smithi169.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-21T19:45:16.181 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-21T19:45:16.181 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-21T19:45:16.182 INFO:teuthology.orchestra.run.smithi169.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-21T19:45:16.182 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-21T19:45:16.182 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:45:16.182 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-21T19:45:16.182 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:45:16.182 INFO:teuthology.orchestra.run.smithi169.stdout:Install 6 Packages 2024-01-21T19:45:16.182 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:45:16.182 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 53 M 2024-01-21T19:45:16.183 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 229 M 2024-01-21T19:45:16.183 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-21T19:45:16.623 INFO:teuthology.orchestra.run.smithi169.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 873 kB/s | 169 kB 00:00 2024-01-21T19:45:17.165 INFO:teuthology.orchestra.run.smithi169.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 71 MB/s | 52 MB 00:00 2024-01-21T19:45:17.182 INFO:teuthology.orchestra.run.smithi169.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 270 kB/s | 203 kB 00:00 2024-01-21T19:45:17.207 INFO:teuthology.orchestra.run.smithi169.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 322 kB/s | 188 kB 00:00 2024-01-21T19:45:17.215 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:45:17.217 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:45:17.217 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-01-21T19:45:17.217 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:45:17.217 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-01-21T19:45:17.217 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-21T19:45:17.217 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-01-21T19:45:17.217 INFO:teuthology.orchestra.run.smithi173.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-21T19:45:17.217 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-21T19:45:17.217 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-21T19:45:17.217 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-21T19:45:17.217 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-21T19:45:17.217 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:45:17.217 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-01-21T19:45:17.218 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:45:17.218 INFO:teuthology.orchestra.run.smithi173.stdout:Install 6 Packages 2024-01-21T19:45:17.218 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:45:17.218 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 53 M 2024-01-21T19:45:17.218 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 229 M 2024-01-21T19:45:17.218 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-01-21T19:45:17.299 INFO:teuthology.orchestra.run.smithi169.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.4 MB/s | 323 kB 00:00 2024-01-21T19:45:17.333 INFO:teuthology.orchestra.run.smithi169.stdout:(6/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 470 kB/s | 71 kB 00:00 2024-01-21T19:45:17.334 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:45:17.334 INFO:teuthology.orchestra.run.smithi169.stdout:Total 46 MB/s | 53 MB 00:01 2024-01-21T19:45:17.347 INFO:teuthology.orchestra.run.smithi173.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 1.9 MB/s | 169 kB 00:00 2024-01-21T19:45:17.376 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-21T19:45:17.398 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-21T19:45:17.398 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-21T19:45:17.455 INFO:teuthology.orchestra.run.smithi173.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 1.0 MB/s | 203 kB 00:00 2024-01-21T19:45:17.564 INFO:teuthology.orchestra.run.smithi173.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 866 kB/s | 188 kB 00:00 2024-01-21T19:45:17.598 INFO:teuthology.orchestra.run.smithi173.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.2 MB/s | 323 kB 00:00 2024-01-21T19:45:17.623 INFO:teuthology.orchestra.run.smithi173.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-01-21T19:45:17.712 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-21T19:45:17.716 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-21T19:45:18.123 INFO:teuthology.orchestra.run.smithi173.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 60 MB/s | 52 MB 00:00 2024-01-21T19:45:18.125 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:45:18.126 INFO:teuthology.orchestra.run.smithi173.stdout:Total 58 MB/s | 53 MB 00:00 2024-01-21T19:45:18.168 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-01-21T19:45:18.189 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-01-21T19:45:18.189 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-01-21T19:45:18.199 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-21T19:45:18.423 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-21T19:45:18.504 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-01-21T19:45:18.507 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-01-21T19:45:18.606 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-21T19:45:18.753 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T19:45:18.847 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T19:45:19.018 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-21T19:45:19.019 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-01-21T19:45:19.170 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T19:45:19.234 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-21T19:45:19.425 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-21T19:45:19.622 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T19:45:19.703 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T19:45:19.879 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-21T19:45:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T19:45:24.049 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T19:45:24.284 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T19:45:24.576 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T19:45:24.577 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-21T19:45:24.577 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-21T19:45:24.577 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-21T19:45:24.577 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-21T19:45:24.577 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-21T19:45:24.907 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T19:45:25.019 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-21T19:45:25.019 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:45:25.019 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-21T19:45:25.019 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-21T19:45:25.019 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-21T19:45:25.019 INFO:teuthology.orchestra.run.smithi169.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-21T19:45:25.019 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:45:25.020 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:45:25.162 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T19:45:25.239 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph 2024-01-21T19:45:25.458 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T19:45:25.459 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-21T19:45:25.459 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-21T19:45:25.459 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-21T19:45:25.459 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-21T19:45:25.459 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-21T19:45:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-21T19:45:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:45:25.908 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-01-21T19:45:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-21T19:45:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-21T19:45:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-21T19:45:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:45:25.909 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:45:25.970 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:52 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:45:26.196 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph 2024-01-21T19:45:26.924 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:53 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:45:27.454 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:45:27.457 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:45:27.457 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-01-21T19:45:27.457 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:45:27.457 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-21T19:45:27.457 INFO:teuthology.orchestra.run.smithi169.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-21T19:45:27.458 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-01-21T19:45:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-21T19:45:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-21T19:45:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-21T19:45:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-21T19:45:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-21T19:45:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-21T19:45:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-21T19:45:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-21T19:45:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-21T19:45:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-21T19:45:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-21T19:45:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-21T19:45:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-21T19:45:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-21T19:45:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-21T19:45:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-21T19:45:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-21T19:45:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-21T19:45:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-21T19:45:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-21T19:45:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-21T19:45:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-21T19:45:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-21T19:45:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-21T19:45:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-21T19:45:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-21T19:45:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-21T19:45:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-21T19:45:27.460 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-21T19:45:27.460 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-21T19:45:27.460 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-21T19:45:27.460 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:45:27.460 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-21T19:45:27.460 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:45:27.460 INFO:teuthology.orchestra.run.smithi169.stdout:Install 32 Packages 2024-01-21T19:45:27.460 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:45:27.460 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 31 M 2024-01-21T19:45:27.461 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 111 M 2024-01-21T19:45:27.461 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-21T19:45:27.926 INFO:teuthology.orchestra.run.smithi169.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 105 kB/s | 6.3 kB 00:00 2024-01-21T19:45:27.976 INFO:teuthology.orchestra.run.smithi169.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-01-21T19:45:28.027 INFO:teuthology.orchestra.run.smithi169.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-01-21T19:45:28.061 INFO:teuthology.orchestra.run.smithi169.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2024-01-21T19:45:28.144 INFO:teuthology.orchestra.run.smithi169.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 23 MB/s | 5.0 MB 00:00 2024-01-21T19:45:28.369 INFO:teuthology.orchestra.run.smithi169.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 46 MB/s | 18 MB 00:00 2024-01-21T19:45:28.385 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:45:28.388 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:45:28.388 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-01-21T19:45:28.388 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:45:28.388 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-01-21T19:45:28.388 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-21T19:45:28.388 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-01-21T19:45:28.389 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-21T19:45:28.389 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-21T19:45:28.389 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-21T19:45:28.389 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-21T19:45:28.389 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-21T19:45:28.389 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-21T19:45:28.389 INFO:teuthology.orchestra.run.smithi173.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-21T19:45:28.389 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-21T19:45:28.389 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-21T19:45:28.389 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-21T19:45:28.389 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-21T19:45:28.389 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-21T19:45:28.389 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-21T19:45:28.389 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-21T19:45:28.390 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-21T19:45:28.390 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-21T19:45:28.390 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-21T19:45:28.390 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-21T19:45:28.390 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-21T19:45:28.390 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-21T19:45:28.390 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-21T19:45:28.390 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-21T19:45:28.390 INFO:teuthology.orchestra.run.smithi173.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-21T19:45:28.390 INFO:teuthology.orchestra.run.smithi173.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-21T19:45:28.390 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-21T19:45:28.390 INFO:teuthology.orchestra.run.smithi173.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-21T19:45:28.390 INFO:teuthology.orchestra.run.smithi173.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-21T19:45:28.390 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-21T19:45:28.390 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-21T19:45:28.391 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-21T19:45:28.391 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-21T19:45:28.391 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:45:28.391 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-01-21T19:45:28.391 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:45:28.391 INFO:teuthology.orchestra.run.smithi173.stdout:Install 32 Packages 2024-01-21T19:45:28.391 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:45:28.391 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 31 M 2024-01-21T19:45:28.391 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 111 M 2024-01-21T19:45:28.392 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-01-21T19:45:28.394 INFO:teuthology.orchestra.run.smithi169.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 155 kB/s | 39 kB 00:00 2024-01-21T19:45:28.419 INFO:teuthology.orchestra.run.smithi169.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 437 kB/s | 157 kB 00:00 2024-01-21T19:45:28.478 INFO:teuthology.orchestra.run.smithi169.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 681 kB/s | 57 kB 00:00 2024-01-21T19:45:28.537 INFO:teuthology.orchestra.run.smithi169.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.8 MB/s | 457 kB 00:00 2024-01-21T19:45:28.562 INFO:teuthology.orchestra.run.smithi169.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 537 kB/s | 103 kB 00:00 2024-01-21T19:45:28.746 INFO:teuthology.orchestra.run.smithi169.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 931 kB/s | 249 kB 00:00 2024-01-21T19:45:28.771 INFO:teuthology.orchestra.run.smithi169.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 522 kB/s | 109 kB 00:00 2024-01-21T19:45:28.847 INFO:teuthology.orchestra.run.smithi169.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 2.0 MB/s | 641 kB 00:00 2024-01-21T19:45:28.913 INFO:teuthology.orchestra.run.smithi169.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 261 kB/s | 44 kB 00:00 2024-01-21T19:45:28.964 INFO:teuthology.orchestra.run.smithi169.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 965 kB/s | 185 kB 00:00 2024-01-21T19:45:28.992 INFO:teuthology.orchestra.run.smithi173.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 111 kB/s | 6.3 kB 00:00 2024-01-21T19:45:29.006 INFO:teuthology.orchestra.run.smithi169.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.1 MB/s | 173 kB 00:00 2024-01-21T19:45:29.022 INFO:teuthology.orchestra.run.smithi169.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 189 kB/s | 11 kB 00:00 2024-01-21T19:45:29.039 INFO:teuthology.orchestra.run.smithi169.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 525 kB/s | 18 kB 00:00 2024-01-21T19:45:29.042 INFO:teuthology.orchestra.run.smithi173.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-01-21T19:45:29.056 INFO:teuthology.orchestra.run.smithi169.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 1.4 MB/s | 49 kB 00:00 2024-01-21T19:45:29.073 INFO:teuthology.orchestra.run.smithi169.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.7 MB/s | 59 kB 00:00 2024-01-21T19:45:29.092 INFO:teuthology.orchestra.run.smithi173.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-01-21T19:45:29.098 INFO:teuthology.orchestra.run.smithi169.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.0 MB/s | 384 kB 00:00 2024-01-21T19:45:29.116 INFO:teuthology.orchestra.run.smithi169.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 373 kB/s | 16 kB 00:00 2024-01-21T19:45:29.118 INFO:teuthology.orchestra.run.smithi173.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2024-01-21T19:45:29.132 INFO:teuthology.orchestra.run.smithi169.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 556 kB/s | 19 kB 00:00 2024-01-21T19:45:29.149 INFO:teuthology.orchestra.run.smithi169.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 712 kB/s | 24 kB 00:00 2024-01-21T19:45:29.175 INFO:teuthology.orchestra.run.smithi169.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 698 kB/s | 29 kB 00:00 2024-01-21T19:45:29.200 INFO:teuthology.orchestra.run.smithi169.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 547 kB/s | 27 kB 00:00 2024-01-21T19:45:29.201 INFO:teuthology.orchestra.run.smithi173.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 24 MB/s | 5.0 MB 00:00 2024-01-21T19:45:29.225 INFO:teuthology.orchestra.run.smithi169.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.6 MB/s | 283 kB 00:00 2024-01-21T19:45:29.267 INFO:teuthology.orchestra.run.smithi169.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.7 MB/s | 255 kB 00:00 2024-01-21T19:45:29.293 INFO:teuthology.orchestra.run.smithi169.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.7 MB/s | 251 kB 00:00 2024-01-21T19:45:29.326 INFO:teuthology.orchestra.run.smithi169.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 850 kB/s | 85 kB 00:00 2024-01-21T19:45:29.351 INFO:teuthology.orchestra.run.smithi169.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 274 kB/s | 23 kB 00:00 2024-01-21T19:45:29.356 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:45:29.356 INFO:teuthology.orchestra.run.smithi169.stdout:Total 16 MB/s | 31 MB 00:01 2024-01-21T19:45:29.437 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-21T19:45:29.460 INFO:teuthology.orchestra.run.smithi173.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 44 MB/s | 18 MB 00:00 2024-01-21T19:45:29.476 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-21T19:45:29.476 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-21T19:45:29.485 INFO:teuthology.orchestra.run.smithi173.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 137 kB/s | 39 kB 00:00 2024-01-21T19:45:29.502 INFO:teuthology.orchestra.run.smithi173.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 409 kB/s | 157 kB 00:00 2024-01-21T19:45:29.569 INFO:teuthology.orchestra.run.smithi173.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 681 kB/s | 57 kB 00:00 2024-01-21T19:45:29.620 INFO:teuthology.orchestra.run.smithi173.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.8 MB/s | 457 kB 00:00 2024-01-21T19:45:29.645 INFO:teuthology.orchestra.run.smithi173.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 561 kB/s | 103 kB 00:00 2024-01-21T19:45:29.792 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-21T19:45:29.796 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-21T19:45:29.837 INFO:teuthology.orchestra.run.smithi173.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 930 kB/s | 249 kB 00:00 2024-01-21T19:45:29.862 INFO:teuthology.orchestra.run.smithi173.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 502 kB/s | 109 kB 00:00 2024-01-21T19:45:29.954 INFO:teuthology.orchestra.run.smithi173.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.9 MB/s | 641 kB 00:00 2024-01-21T19:45:30.013 INFO:teuthology.orchestra.run.smithi173.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 248 kB/s | 44 kB 00:00 2024-01-21T19:45:30.038 INFO:teuthology.orchestra.run.smithi173.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.0 MB/s | 185 kB 00:00 2024-01-21T19:45:30.080 INFO:teuthology.orchestra.run.smithi173.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 267 kB/s | 11 kB 00:00 2024-01-21T19:45:30.122 INFO:teuthology.orchestra.run.smithi173.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 422 kB/s | 18 kB 00:00 2024-01-21T19:45:30.147 INFO:teuthology.orchestra.run.smithi173.stdout:(19/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 896 kB/s | 173 kB 00:00 2024-01-21T19:45:30.172 INFO:teuthology.orchestra.run.smithi173.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.4 MB/s | 384 kB 00:00 2024-01-21T19:45:30.189 INFO:teuthology.orchestra.run.smithi173.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 725 kB/s | 49 kB 00:00 2024-01-21T19:45:30.206 INFO:teuthology.orchestra.run.smithi173.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 999 kB/s | 59 kB 00:00 2024-01-21T19:45:30.264 INFO:teuthology.orchestra.run.smithi173.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 209 kB/s | 16 kB 00:00 2024-01-21T19:45:30.298 INFO:teuthology.orchestra.run.smithi173.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 203 kB/s | 19 kB 00:00 2024-01-21T19:45:30.324 INFO:teuthology.orchestra.run.smithi173.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 404 kB/s | 24 kB 00:00 2024-01-21T19:45:30.349 INFO:teuthology.orchestra.run.smithi173.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 581 kB/s | 29 kB 00:00 2024-01-21T19:45:30.374 INFO:teuthology.orchestra.run.smithi173.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 548 kB/s | 27 kB 00:00 2024-01-21T19:45:30.399 INFO:teuthology.orchestra.run.smithi173.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2024-01-21T19:45:30.425 INFO:teuthology.orchestra.run.smithi173.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2024-01-21T19:45:30.459 INFO:teuthology.orchestra.run.smithi173.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.4 MB/s | 85 kB 00:00 2024-01-21T19:45:30.484 INFO:teuthology.orchestra.run.smithi173.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 392 kB/s | 23 kB 00:00 2024-01-21T19:45:30.509 INFO:teuthology.orchestra.run.smithi173.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.8 MB/s | 251 kB 00:00 2024-01-21T19:45:30.513 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:45:30.513 INFO:teuthology.orchestra.run.smithi173.stdout:Total 15 MB/s | 31 MB 00:02 2024-01-21T19:45:30.590 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-01-21T19:45:30.628 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-01-21T19:45:30.629 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-01-21T19:45:30.903 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-21T19:45:30.956 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-01-21T19:45:30.959 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-01-21T19:45:31.097 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-21T19:45:31.300 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-21T19:45:31.558 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-21T19:45:31.749 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-21T19:45:31.921 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-21T19:45:32.084 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-21T19:45:32.085 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-01-21T19:45:32.279 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-21T19:45:32.309 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-21T19:45:32.448 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-21T19:45:32.512 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-21T19:45:32.649 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-21T19:45:32.794 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-21T19:45:32.852 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-21T19:45:32.993 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-21T19:45:33.057 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-21T19:45:33.191 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-21T19:45:33.348 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-21T19:45:33.363 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-21T19:45:33.506 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-21T19:45:33.693 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-21T19:45:33.722 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-21T19:45:33.869 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-21T19:45:33.970 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-21T19:45:34.114 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-21T19:45:34.175 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-21T19:45:34.327 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-21T19:45:34.417 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-21T19:45:34.593 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T19:45:34.609 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-21T19:45:34.892 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-21T19:45:34.970 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-21T19:45:35.147 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-21T19:45:35.240 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-21T19:45:35.311 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-21T19:45:35.411 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-21T19:45:35.530 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-21T19:45:35.681 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-21T19:45:35.781 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-21T19:45:35.854 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T19:45:36.055 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-21T19:45:36.168 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-21T19:45:36.293 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-21T19:45:36.376 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-21T19:45:36.599 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-21T19:45:36.625 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-21T19:45:36.831 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-21T19:45:36.956 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-21T19:45:37.108 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-21T19:45:37.323 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-21T19:45:37.364 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-21T19:45:37.566 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-21T19:45:37.576 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T19:45:37.904 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-21T19:45:38.201 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-21T19:45:38.581 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-21T19:45:38.776 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T19:45:39.114 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T19:45:39.317 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T19:45:39.752 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T19:45:39.984 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T19:45:40.207 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T19:45:40.311 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T19:45:40.400 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T19:45:40.417 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T19:45:40.509 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T19:45:40.557 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T19:45:40.943 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T19:45:40.962 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T19:45:40.962 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-21T19:45:40.962 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-21T19:45:40.963 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-21T19:45:40.963 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-21T19:45:40.963 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-21T19:45:40.963 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-21T19:45:40.963 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-21T19:45:40.963 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-21T19:45:40.964 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-21T19:45:40.964 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-21T19:45:40.964 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-21T19:45:40.964 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-21T19:45:40.964 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-21T19:45:40.964 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-21T19:45:40.965 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-21T19:45:40.965 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-21T19:45:40.965 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T19:45:40.965 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-21T19:45:40.965 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-21T19:45:40.965 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-21T19:45:40.966 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-21T19:45:40.966 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-21T19:45:40.966 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-21T19:45:40.966 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-21T19:45:40.966 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-21T19:45:40.966 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-21T19:45:40.966 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-21T19:45:40.967 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-21T19:45:40.967 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-21T19:45:40.967 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-21T19:45:40.967 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-21T19:45:41.183 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T19:45:41.406 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T19:45:41.586 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T19:45:41.604 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T19:45:41.711 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-21T19:45:41.712 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:45:41.712 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-21T19:45:41.712 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:41.712 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:41.712 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:41.712 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-21T19:45:41.712 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:41.712 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:41.712 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-21T19:45:41.712 INFO:teuthology.orchestra.run.smithi169.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-21T19:45:41.712 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-21T19:45:41.712 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-21T19:45:41.713 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-21T19:45:41.713 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-21T19:45:41.713 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-21T19:45:41.713 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-21T19:45:41.713 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-21T19:45:41.713 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-21T19:45:41.713 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-21T19:45:41.713 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-21T19:45:41.713 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-21T19:45:41.713 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend-2.6-1.el8.noarch 2024-01-21T19:45:41.713 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-21T19:45:41.713 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-21T19:45:41.713 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-21T19:45:41.713 INFO:teuthology.orchestra.run.smithi169.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-21T19:45:41.713 INFO:teuthology.orchestra.run.smithi169.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-21T19:45:41.714 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-21T19:45:41.714 INFO:teuthology.orchestra.run.smithi169.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-21T19:45:41.714 INFO:teuthology.orchestra.run.smithi169.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-21T19:45:41.714 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-21T19:45:41.714 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-21T19:45:41.714 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-21T19:45:41.714 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-21T19:45:41.714 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:45:41.714 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:45:41.752 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T19:45:41.931 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-base 2024-01-21T19:45:42.146 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T19:45:42.146 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-21T19:45:42.146 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-21T19:45:42.146 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-21T19:45:42.146 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-21T19:45:42.146 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-21T19:45:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-21T19:45:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-21T19:45:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-21T19:45:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-21T19:45:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-21T19:45:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-21T19:45:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-21T19:45:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-21T19:45:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-21T19:45:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-21T19:45:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-21T19:45:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T19:45:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-21T19:45:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-21T19:45:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-21T19:45:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-21T19:45:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-21T19:45:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-21T19:45:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-21T19:45:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-21T19:45:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-21T19:45:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-21T19:45:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-21T19:45:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-21T19:45:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-21T19:45:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-21T19:45:42.663 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:09 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:45:43.067 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-21T19:45:43.068 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:45:43.068 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-01-21T19:45:43.068 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:43.068 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:43.068 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:43.068 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-21T19:45:43.068 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:43.068 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:43.068 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-21T19:45:43.068 INFO:teuthology.orchestra.run.smithi173.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-21T19:45:43.068 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-21T19:45:43.068 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-21T19:45:43.069 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-21T19:45:43.069 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-21T19:45:43.069 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-21T19:45:43.069 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-21T19:45:43.069 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-21T19:45:43.069 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-21T19:45:43.069 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-21T19:45:43.069 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-21T19:45:43.069 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-21T19:45:43.069 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-2.6-1.el8.noarch 2024-01-21T19:45:43.069 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-21T19:45:43.069 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-21T19:45:43.069 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-21T19:45:43.069 INFO:teuthology.orchestra.run.smithi173.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-21T19:45:43.069 INFO:teuthology.orchestra.run.smithi173.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-21T19:45:43.070 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-21T19:45:43.070 INFO:teuthology.orchestra.run.smithi173.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-21T19:45:43.070 INFO:teuthology.orchestra.run.smithi173.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-21T19:45:43.070 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-21T19:45:43.070 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-21T19:45:43.070 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-21T19:45:43.070 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-21T19:45:43.070 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:45:43.070 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:45:43.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-base 2024-01-21T19:45:44.059 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:11 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:45:44.106 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:45:44.152 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:45:44.153 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-01-21T19:45:44.153 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:45:44.246 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install cephadm 2024-01-21T19:45:44.971 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:11 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:45:45.492 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:45:45.538 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:45:45.539 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-01-21T19:45:45.539 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:45:45.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install cephadm 2024-01-21T19:45:46.384 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:13 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:45:46.468 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:45:46.469 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:45:46.469 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-01-21T19:45:46.469 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:45:46.469 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-21T19:45:46.469 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-21T19:45:46.469 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:45:46.469 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-21T19:45:46.469 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:45:46.469 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-01-21T19:45:46.469 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:45:46.469 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 71 k 2024-01-21T19:45:46.469 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 290 k 2024-01-21T19:45:46.470 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-21T19:45:46.515 INFO:teuthology.orchestra.run.smithi169.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.5 MB/s | 71 kB 00:00 2024-01-21T19:45:46.515 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:45:46.515 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.5 MB/s | 71 kB 00:00 2024-01-21T19:45:46.515 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-21T19:45:46.522 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-21T19:45:46.522 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-21T19:45:46.524 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-21T19:45:46.527 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-21T19:45:46.569 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-21T19:45:47.076 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:45:47.237 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:45:47.474 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:45:47.854 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:45:47.855 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:45:47.855 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-01-21T19:45:47.855 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:45:47.856 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-01-21T19:45:47.856 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-21T19:45:47.856 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:45:47.856 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-01-21T19:45:47.856 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:45:47.856 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-01-21T19:45:47.856 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:45:47.856 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 71 k 2024-01-21T19:45:47.856 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 290 k 2024-01-21T19:45:47.856 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-01-21T19:45:47.892 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:45:47.892 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:45:47.892 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-21T19:45:47.892 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-21T19:45:47.892 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:45:47.892 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:45:47.898 INFO:teuthology.orchestra.run.smithi173.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.7 MB/s | 71 kB 00:00 2024-01-21T19:45:47.898 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:45:47.899 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.6 MB/s | 71 kB 00:00 2024-01-21T19:45:47.899 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-01-21T19:45:47.905 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-01-21T19:45:47.905 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-01-21T19:45:47.907 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-01-21T19:45:47.910 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-01-21T19:45:47.957 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-01-21T19:45:48.119 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-immutable-object-cache 2024-01-21T19:45:48.459 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:45:48.620 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:45:48.851 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:45:48.852 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:15 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:45:49.218 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:45:49.218 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:45:49.218 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-01-21T19:45:49.218 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-21T19:45:49.218 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:45:49.218 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:45:49.451 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-immutable-object-cache 2024-01-21T19:45:50.188 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:17 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:45:50.334 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:45:50.335 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:45:50.335 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-01-21T19:45:50.336 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:45:50.336 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-21T19:45:50.336 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-21T19:45:50.336 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:45:50.336 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-21T19:45:50.337 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:45:50.337 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-01-21T19:45:50.337 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:45:50.337 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 163 k 2024-01-21T19:45:50.337 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 587 k 2024-01-21T19:45:50.337 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-21T19:45:50.386 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.2 MB/s | 163 kB 00:00 2024-01-21T19:45:50.387 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:45:50.388 INFO:teuthology.orchestra.run.smithi169.stdout:Total 3.1 MB/s | 163 kB 00:00 2024-01-21T19:45:50.388 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-21T19:45:50.398 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-21T19:45:50.398 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-21T19:45:50.446 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-21T19:45:50.449 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-21T19:45:50.557 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-21T19:45:51.675 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:45:51.676 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:45:51.676 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-01-21T19:45:51.676 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:45:51.676 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-01-21T19:45:51.676 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-21T19:45:51.676 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:45:51.676 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-01-21T19:45:51.676 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:45:51.677 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-01-21T19:45:51.677 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:45:51.677 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 163 k 2024-01-21T19:45:51.677 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 587 k 2024-01-21T19:45:51.677 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-01-21T19:45:51.719 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.8 MB/s | 163 kB 00:00 2024-01-21T19:45:51.719 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:45:51.720 INFO:teuthology.orchestra.run.smithi173.stdout:Total 3.7 MB/s | 163 kB 00:00 2024-01-21T19:45:51.720 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-01-21T19:45:51.731 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-01-21T19:45:51.731 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-01-21T19:45:51.778 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-01-21T19:45:51.781 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-01-21T19:45:51.908 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:45:52.069 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-01-21T19:45:52.347 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:45:52.730 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:45:52.730 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:45:52.730 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-21T19:45:52.730 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:52.730 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:45:52.730 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:45:52.954 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr 2024-01-21T19:45:53.409 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:45:53.689 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:20 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:45:53.830 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:45:54.184 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:45:54.184 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:45:54.184 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-01-21T19:45:54.184 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-21T19:45:54.184 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:45:54.184 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:45:54.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr 2024-01-21T19:45:55.131 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:45:55.161 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:22 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:45:55.177 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:45:55.178 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-01-21T19:45:55.178 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:45:55.275 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-dashboard 2024-01-21T19:45:55.998 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:23 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:45:56.596 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:45:56.642 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:45:56.643 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-01-21T19:45:56.643 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:45:56.756 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-dashboard 2024-01-21T19:45:57.480 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:24 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:45:57.489 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:45:57.491 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:45:57.491 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-01-21T19:45:57.491 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:45:57.491 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-21T19:45:57.491 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-21T19:45:57.491 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-01-21T19:45:57.491 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-21T19:45:57.491 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-21T19:45:57.492 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-21T19:45:57.492 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-21T19:45:57.492 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-21T19:45:57.492 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:45:57.492 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-21T19:45:57.492 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:45:57.492 INFO:teuthology.orchestra.run.smithi169.stdout:Install 6 Packages 2024-01-21T19:45:57.492 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:45:57.493 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 3.0 M 2024-01-21T19:45:57.493 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 15 M 2024-01-21T19:45:57.493 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-21T19:45:58.131 INFO:teuthology.orchestra.run.smithi169.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 114 kB/s | 24 kB 00:00 2024-01-21T19:45:58.156 INFO:teuthology.orchestra.run.smithi169.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 38 kB/s | 8.9 kB 00:00 2024-01-21T19:45:58.215 INFO:teuthology.orchestra.run.smithi169.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 9.0 MB/s | 2.7 MB 00:00 2024-01-21T19:45:58.340 INFO:teuthology.orchestra.run.smithi169.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 204 kB/s | 43 kB 00:00 2024-01-21T19:45:58.399 INFO:teuthology.orchestra.run.smithi169.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 139 kB/s | 34 kB 00:00 2024-01-21T19:45:58.849 INFO:teuthology.orchestra.run.smithi169.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 310 kB/s | 196 kB 00:00 2024-01-21T19:45:58.851 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:45:58.851 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.2 MB/s | 3.0 MB 00:01 2024-01-21T19:45:58.863 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-21T19:45:58.873 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-21T19:45:58.873 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-21T19:45:58.946 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:45:58.947 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:45:58.947 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-01-21T19:45:58.947 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:45:58.947 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-01-21T19:45:58.947 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-21T19:45:58.948 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-01-21T19:45:58.948 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-21T19:45:58.948 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-21T19:45:58.948 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-21T19:45:58.948 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-21T19:45:58.948 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-21T19:45:58.948 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:45:58.948 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-01-21T19:45:58.948 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:45:58.948 INFO:teuthology.orchestra.run.smithi173.stdout:Install 6 Packages 2024-01-21T19:45:58.948 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:45:58.949 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 3.0 M 2024-01-21T19:45:58.949 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 15 M 2024-01-21T19:45:58.949 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-01-21T19:45:59.048 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-21T19:45:59.051 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-21T19:45:59.311 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-21T19:45:59.527 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-21T19:45:59.548 INFO:teuthology.orchestra.run.smithi173.stdout:(1/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 161 kB/s | 8.9 kB 00:00 2024-01-21T19:45:59.565 INFO:teuthology.orchestra.run.smithi173.stdout:(2/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 331 kB/s | 24 kB 00:00 2024-01-21T19:45:59.623 INFO:teuthology.orchestra.run.smithi173.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 20 MB/s | 2.7 MB 00:00 2024-01-21T19:45:59.699 INFO:teuthology.orchestra.run.smithi173.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 252 kB/s | 34 kB 00:00 2024-01-21T19:45:59.732 INFO:teuthology.orchestra.run.smithi173.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 231 kB/s | 43 kB 00:00 2024-01-21T19:45:59.780 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-21T19:45:59.791 INFO:teuthology.orchestra.run.smithi173.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.1 MB/s | 196 kB 00:00 2024-01-21T19:45:59.793 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:45:59.793 INFO:teuthology.orchestra.run.smithi173.stdout:Total 3.5 MB/s | 3.0 MB 00:00 2024-01-21T19:45:59.802 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-01-21T19:45:59.812 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-01-21T19:45:59.812 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-01-21T19:45:59.964 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-21T19:45:59.966 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-01-21T19:45:59.969 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-01-21T19:46:00.085 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-21T19:46:00.230 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-01-21T19:46:00.472 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-21T19:46:00.716 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-21T19:46:00.926 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-21T19:46:01.047 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-21T19:46:01.109 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-21T19:46:01.388 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:46:01.684 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:46:01.684 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-21T19:46:01.684 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-21T19:46:01.684 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-21T19:46:01.684 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-21T19:46:01.684 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-21T19:46:02.063 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-21T19:46:02.088 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-21T19:46:02.088 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:02.088 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-21T19:46:02.088 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-21T19:46:02.088 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-21T19:46:02.088 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-21T19:46:02.088 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-21T19:46:02.088 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-21T19:46:02.088 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-21T19:46:02.088 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:02.089 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:46:02.208 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-21T19:46:02.355 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:46:02.639 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:46:02.640 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-21T19:46:02.640 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-21T19:46:02.640 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-21T19:46:02.640 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-21T19:46:02.640 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-21T19:46:02.940 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:29 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:03.033 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-21T19:46:03.033 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:03.033 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-01-21T19:46:03.033 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-21T19:46:03.034 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-21T19:46:03.034 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-21T19:46:03.034 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-21T19:46:03.034 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-21T19:46:03.034 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-21T19:46:03.034 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:03.034 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:46:03.197 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-21T19:46:03.920 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:30 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:04.410 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:46:04.411 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-01-21T19:46:04.411 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-01-21T19:46:04.411 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-01-21T19:46:04.411 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-21T19:46:04.412 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-21T19:46:04.412 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-01-21T19:46:04.412 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-21T19:46:04.412 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-21T19:46:04.412 INFO:teuthology.orchestra.run.smithi169.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-21T19:46:04.412 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-21T19:46:04.412 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-21T19:46:04.412 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-21T19:46:04.412 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-21T19:46:04.412 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:04.412 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-21T19:46:04.412 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-01-21T19:46:04.412 INFO:teuthology.orchestra.run.smithi169.stdout:Install 8 Packages 2024-01-21T19:46:04.412 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:04.413 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 37 M 2024-01-21T19:46:04.413 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 210 M 2024-01-21T19:46:04.413 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-21T19:46:04.796 INFO:teuthology.orchestra.run.smithi169.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 55 MB/s | 8.6 MB 00:00 2024-01-21T19:46:05.046 INFO:teuthology.orchestra.run.smithi169.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2024-01-21T19:46:05.121 INFO:teuthology.orchestra.run.smithi169.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 10 MB/s | 4.9 MB 00:00 2024-01-21T19:46:05.146 INFO:teuthology.orchestra.run.smithi169.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.2 MB/s | 225 kB 00:00 2024-01-21T19:46:05.313 INFO:teuthology.orchestra.run.smithi169.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 7.2 MB/s | 3.7 MB 00:00 2024-01-21T19:46:05.421 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:46:05.421 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-01-21T19:46:05.421 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-01-21T19:46:05.421 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-01-21T19:46:05.421 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-01-21T19:46:05.421 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-21T19:46:05.421 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-01-21T19:46:05.421 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-21T19:46:05.421 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-21T19:46:05.422 INFO:teuthology.orchestra.run.smithi173.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-21T19:46:05.422 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-21T19:46:05.422 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-21T19:46:05.422 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-21T19:46:05.422 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-21T19:46:05.422 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:05.422 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-01-21T19:46:05.422 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-01-21T19:46:05.422 INFO:teuthology.orchestra.run.smithi173.stdout:Install 8 Packages 2024-01-21T19:46:05.422 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:05.422 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 37 M 2024-01-21T19:46:05.422 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 210 M 2024-01-21T19:46:05.423 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-01-21T19:46:05.581 INFO:teuthology.orchestra.run.smithi169.stdout:(6/8): python3-scipy-1.0.0-21.module_el8+762+77 31 MB/s | 14 MB 00:00 2024-01-21T19:46:05.606 INFO:teuthology.orchestra.run.smithi169.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 591 kB/s | 172 kB 00:00 2024-01-21T19:46:05.631 INFO:teuthology.orchestra.run.smithi169.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.3 MB/s | 645 kB 00:00 2024-01-21T19:46:05.631 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:46:05.631 INFO:teuthology.orchestra.run.smithi169.stdout:Total 30 MB/s | 37 MB 00:01 2024-01-21T19:46:05.804 INFO:teuthology.orchestra.run.smithi173.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 58 MB/s | 8.6 MB 00:00 2024-01-21T19:46:05.861 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-21T19:46:05.880 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-21T19:46:05.880 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-21T19:46:06.087 INFO:teuthology.orchestra.run.smithi173.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 11 MB/s | 4.7 MB 00:00 2024-01-21T19:46:06.155 INFO:teuthology.orchestra.run.smithi173.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 9.7 MB/s | 4.9 MB 00:00 2024-01-21T19:46:06.180 INFO:teuthology.orchestra.run.smithi173.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2024-01-21T19:46:06.210 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-21T19:46:06.213 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-21T19:46:06.438 INFO:teuthology.orchestra.run.smithi173.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 5.9 MB/s | 3.7 MB 00:00 2024-01-21T19:46:06.656 INFO:teuthology.orchestra.run.smithi173.stdout:(6/8): python3-scipy-1.0.0-21.module_el8+762+77 28 MB/s | 14 MB 00:00 2024-01-21T19:46:06.681 INFO:teuthology.orchestra.run.smithi173.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.3 MB/s | 645 kB 00:00 2024-01-21T19:46:06.714 INFO:teuthology.orchestra.run.smithi173.stdout:(8/8): libquadmath-8.5.0-21.el8.x86_64.rpm 626 kB/s | 172 kB 00:00 2024-01-21T19:46:06.715 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:46:06.715 INFO:teuthology.orchestra.run.smithi173.stdout:Total 29 MB/s | 37 MB 00:01 2024-01-21T19:46:06.830 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-21T19:46:06.945 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-01-21T19:46:06.953 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T19:46:06.963 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-01-21T19:46:06.964 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-01-21T19:46:07.075 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T19:46:07.211 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T19:46:07.282 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-01-21T19:46:07.286 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-01-21T19:46:07.803 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T19:46:07.931 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-01-21T19:46:07.978 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T19:46:08.073 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T19:46:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T19:46:08.381 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T19:46:08.592 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T19:46:08.976 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T19:46:09.156 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T19:46:09.444 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T19:46:09.982 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T19:46:10.137 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T19:46:10.439 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T19:46:10.448 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T19:46:10.669 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T19:46:11.123 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T19:46:11.475 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T19:46:11.739 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T19:46:12.440 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T19:46:13.493 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T19:46:13.720 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-21T19:46:13.912 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T19:46:14.224 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T19:46:14.224 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-21T19:46:14.225 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-21T19:46:14.225 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-21T19:46:14.225 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-21T19:46:14.225 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T19:46:14.225 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-21T19:46:14.225 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-21T19:46:14.671 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-21T19:46:14.672 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:14.672 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-21T19:46:14.672 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-21T19:46:14.672 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-21T19:46:14.672 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-21T19:46:14.672 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-21T19:46:14.672 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-21T19:46:14.672 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-21T19:46:14.672 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-21T19:46:14.672 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-21T19:46:14.672 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:14.672 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:46:14.809 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-21T19:46:14.955 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-rook 2024-01-21T19:46:15.037 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T19:46:15.358 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T19:46:15.358 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-21T19:46:15.358 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-21T19:46:15.358 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-21T19:46:15.358 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-21T19:46:15.358 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T19:46:15.358 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-21T19:46:15.358 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-21T19:46:15.680 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:42 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:16.073 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-21T19:46:16.073 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:16.073 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-01-21T19:46:16.074 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-21T19:46:16.074 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-21T19:46:16.074 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-21T19:46:16.074 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-21T19:46:16.074 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-21T19:46:16.074 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-21T19:46:16.074 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-21T19:46:16.074 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-21T19:46:16.074 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:16.074 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:46:16.177 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-rook 2024-01-21T19:46:16.908 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:43 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:17.147 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:46:17.149 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:46:17.149 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-01-21T19:46:17.149 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:46:17.149 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-21T19:46:17.150 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-21T19:46:17.150 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-01-21T19:46:17.150 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-21T19:46:17.150 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-21T19:46:17.150 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-21T19:46:17.150 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-21T19:46:17.150 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-21T19:46:17.150 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-21T19:46:17.150 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-21T19:46:17.150 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-21T19:46:17.151 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-21T19:46:17.151 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-21T19:46:17.151 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-21T19:46:17.151 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-21T19:46:17.151 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:17.151 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-21T19:46:17.151 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:46:17.151 INFO:teuthology.orchestra.run.smithi169.stdout:Install 13 Packages 2024-01-21T19:46:17.151 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:17.152 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 1.9 M 2024-01-21T19:46:17.152 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 24 M 2024-01-21T19:46:17.152 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-21T19:46:17.606 INFO:teuthology.orchestra.run.smithi169.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 584 kB/s | 27 kB 00:00 2024-01-21T19:46:17.689 INFO:teuthology.orchestra.run.smithi169.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 154 kB/s | 20 kB 00:00 2024-01-21T19:46:17.706 INFO:teuthology.orchestra.run.smithi169.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 181 kB/s | 27 kB 00:00 2024-01-21T19:46:17.781 INFO:teuthology.orchestra.run.smithi169.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 720 kB/s | 126 kB 00:00 2024-01-21T19:46:17.806 INFO:teuthology.orchestra.run.smithi169.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 939 kB/s | 110 kB 00:00 2024-01-21T19:46:17.890 INFO:teuthology.orchestra.run.smithi169.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 395 kB/s | 33 kB 00:00 2024-01-21T19:46:17.915 INFO:teuthology.orchestra.run.smithi169.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 740 kB/s | 155 kB 00:00 2024-01-21T19:46:17.941 INFO:teuthology.orchestra.run.smithi169.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 268 kB/s | 43 kB 00:00 2024-01-21T19:46:17.966 INFO:teuthology.orchestra.run.smithi169.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 215 kB/s | 16 kB 00:00 2024-01-21T19:46:18.025 INFO:teuthology.orchestra.run.smithi169.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 746 kB/s | 81 kB 00:00 2024-01-21T19:46:18.075 INFO:teuthology.orchestra.run.smithi169.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 591 kB/s | 64 kB 00:00 2024-01-21T19:46:18.100 INFO:teuthology.orchestra.run.smithi169.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 812 kB/s | 61 kB 00:00 2024-01-21T19:46:18.153 INFO:teuthology.orchestra.run.smithi169.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.5 MB/s | 1.2 MB 00:00 2024-01-21T19:46:18.153 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:46:18.153 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.9 MB/s | 1.9 MB 00:01 2024-01-21T19:46:18.205 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-21T19:46:18.220 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-21T19:46:18.220 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-21T19:46:18.306 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-21T19:46:18.309 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-21T19:46:18.398 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:46:18.400 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:46:18.400 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-01-21T19:46:18.400 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:46:18.400 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-01-21T19:46:18.400 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-21T19:46:18.400 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-01-21T19:46:18.400 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-21T19:46:18.401 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-21T19:46:18.401 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-21T19:46:18.401 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-21T19:46:18.401 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-21T19:46:18.401 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-21T19:46:18.401 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-21T19:46:18.401 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-21T19:46:18.401 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-21T19:46:18.401 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-21T19:46:18.401 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-21T19:46:18.401 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-21T19:46:18.401 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:18.401 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-01-21T19:46:18.401 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:46:18.402 INFO:teuthology.orchestra.run.smithi173.stdout:Install 13 Packages 2024-01-21T19:46:18.402 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:18.402 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 1.9 M 2024-01-21T19:46:18.402 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 24 M 2024-01-21T19:46:18.402 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-01-21T19:46:18.799 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-21T19:46:18.839 INFO:teuthology.orchestra.run.smithi173.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 828 kB/s | 27 kB 00:00 2024-01-21T19:46:18.922 INFO:teuthology.orchestra.run.smithi173.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 172 kB/s | 20 kB 00:00 2024-01-21T19:46:18.939 INFO:teuthology.orchestra.run.smithi173.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 200 kB/s | 27 kB 00:00 2024-01-21T19:46:19.029 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-21T19:46:19.031 INFO:teuthology.orchestra.run.smithi173.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 657 kB/s | 126 kB 00:00 2024-01-21T19:46:19.057 INFO:teuthology.orchestra.run.smithi173.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 821 kB/s | 110 kB 00:00 2024-01-21T19:46:19.157 INFO:teuthology.orchestra.run.smithi173.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 713 kB/s | 155 kB 00:00 2024-01-21T19:46:19.182 INFO:teuthology.orchestra.run.smithi173.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 283 kB/s | 43 kB 00:00 2024-01-21T19:46:19.234 INFO:teuthology.orchestra.run.smithi173.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 186 kB/s | 33 kB 00:00 2024-01-21T19:46:19.248 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-21T19:46:19.325 INFO:teuthology.orchestra.run.smithi173.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 97 kB/s | 16 kB 00:00 2024-01-21T19:46:19.375 INFO:teuthology.orchestra.run.smithi173.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 423 kB/s | 81 kB 00:00 2024-01-21T19:46:19.433 INFO:teuthology.orchestra.run.smithi173.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 592 kB/s | 64 kB 00:00 2024-01-21T19:46:19.459 INFO:teuthology.orchestra.run.smithi173.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 732 kB/s | 61 kB 00:00 2024-01-21T19:46:19.478 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-21T19:46:19.501 INFO:teuthology.orchestra.run.smithi173.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.4 MB/s | 1.2 MB 00:00 2024-01-21T19:46:19.501 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:46:19.501 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.7 MB/s | 1.9 MB 00:01 2024-01-21T19:46:19.559 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-01-21T19:46:19.574 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-01-21T19:46:19.574 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-01-21T19:46:19.662 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-01-21T19:46:19.665 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-01-21T19:46:19.699 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-21T19:46:19.895 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-21T19:46:20.129 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-21T19:46:20.204 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-01-21T19:46:20.448 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-21T19:46:20.504 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-21T19:46:20.644 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-21T19:46:20.721 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-21T19:46:20.906 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-21T19:46:21.253 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-21T19:46:21.452 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-21T19:46:21.691 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-21T19:46:21.775 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-21T19:46:21.975 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-21T19:46:22.032 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-21T19:46:22.241 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-21T19:46:22.250 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-21T19:46:22.408 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-21T19:46:22.559 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T19:46:22.876 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T19:46:22.877 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-21T19:46:22.877 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-21T19:46:22.877 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-21T19:46:22.877 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-21T19:46:22.877 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-21T19:46:22.877 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-21T19:46:22.877 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-21T19:46:22.877 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-21T19:46:22.877 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-21T19:46:22.877 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-21T19:46:22.877 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-21T19:46:22.877 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-21T19:46:23.279 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-21T19:46:23.335 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-21T19:46:23.335 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:23.335 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-21T19:46:23.335 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-21T19:46:23.335 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-21T19:46:23.335 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-21T19:46:23.335 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-21T19:46:23.335 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-21T19:46:23.335 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-21T19:46:23.335 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-21T19:46:23.335 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-21T19:46:23.336 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-21T19:46:23.336 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-21T19:46:23.336 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-21T19:46:23.336 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-21T19:46:23.336 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-21T19:46:23.336 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:23.336 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:46:23.446 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-cephadm 2024-01-21T19:46:23.611 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-21T19:46:23.820 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-21T19:46:24.037 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-21T19:46:24.170 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T19:46:24.173 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:51 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:24.477 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T19:46:24.477 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-21T19:46:24.477 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-21T19:46:24.478 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-21T19:46:24.478 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-21T19:46:24.478 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-21T19:46:24.478 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-21T19:46:24.478 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-21T19:46:24.478 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-21T19:46:24.478 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-21T19:46:24.478 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-21T19:46:24.478 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-21T19:46:24.478 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-21T19:46:24.939 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-21T19:46:24.939 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:24.939 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-01-21T19:46:24.939 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-21T19:46:24.939 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-21T19:46:24.939 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-21T19:46:24.939 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-21T19:46:24.939 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-21T19:46:24.940 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-21T19:46:24.940 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-21T19:46:24.940 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-21T19:46:24.940 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-21T19:46:24.940 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-21T19:46:24.940 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-21T19:46:24.940 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-21T19:46:24.940 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-21T19:46:24.940 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:24.940 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:46:25.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-cephadm 2024-01-21T19:46:25.658 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:46:25.659 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:46:25.659 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-01-21T19:46:25.660 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:46:25.660 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-21T19:46:25.660 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-21T19:46:25.660 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-01-21T19:46:25.660 INFO:teuthology.orchestra.run.smithi169.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-21T19:46:25.660 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-21T19:46:25.660 INFO:teuthology.orchestra.run.smithi169.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-21T19:46:25.660 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-21T19:46:25.660 INFO:teuthology.orchestra.run.smithi169.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-21T19:46:25.660 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:25.660 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-21T19:46:25.660 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:46:25.660 INFO:teuthology.orchestra.run.smithi169.stdout:Install 6 Packages 2024-01-21T19:46:25.661 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:25.661 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 5.8 M 2024-01-21T19:46:25.661 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 24 M 2024-01-21T19:46:25.661 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-21T19:46:25.794 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:52 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:26.034 INFO:teuthology.orchestra.run.smithi169.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.7 MB/s | 89 kB 00:00 2024-01-21T19:46:26.251 INFO:teuthology.orchestra.run.smithi169.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.0 MB/s | 538 kB 00:00 2024-01-21T19:46:26.342 INFO:teuthology.orchestra.run.smithi169.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 69 kB/s | 21 kB 00:00 2024-01-21T19:46:26.493 INFO:teuthology.orchestra.run.smithi169.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 9.4 MB/s | 4.8 MB 00:00 2024-01-21T19:46:26.526 INFO:teuthology.orchestra.run.smithi169.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 266 kB/s | 49 kB 00:00 2024-01-21T19:46:26.777 INFO:teuthology.orchestra.run.smithi169.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 676 kB/s | 355 kB 00:00 2024-01-21T19:46:26.777 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:46:26.777 INFO:teuthology.orchestra.run.smithi169.stdout:Total 5.2 MB/s | 5.8 MB 00:01 2024-01-21T19:46:26.829 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-21T19:46:26.839 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-21T19:46:26.839 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-21T19:46:26.942 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-21T19:46:26.946 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-21T19:46:27.246 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-21T19:46:27.274 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:46:27.275 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:46:27.275 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-01-21T19:46:27.275 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:46:27.275 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-01-21T19:46:27.275 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-21T19:46:27.275 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-01-21T19:46:27.275 INFO:teuthology.orchestra.run.smithi173.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-21T19:46:27.276 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-21T19:46:27.276 INFO:teuthology.orchestra.run.smithi173.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-21T19:46:27.276 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-21T19:46:27.276 INFO:teuthology.orchestra.run.smithi173.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-21T19:46:27.276 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:27.276 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-01-21T19:46:27.276 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:46:27.276 INFO:teuthology.orchestra.run.smithi173.stdout:Install 6 Packages 2024-01-21T19:46:27.276 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:27.277 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 5.8 M 2024-01-21T19:46:27.277 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 24 M 2024-01-21T19:46:27.277 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-01-21T19:46:27.524 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-21T19:46:27.794 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-21T19:46:27.842 INFO:teuthology.orchestra.run.smithi173.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.9 MB/s | 89 kB 00:00 2024-01-21T19:46:28.025 INFO:teuthology.orchestra.run.smithi173.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 116 kB/s | 21 kB 00:00 2024-01-21T19:46:28.075 INFO:teuthology.orchestra.run.smithi173.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.9 MB/s | 538 kB 00:00 2024-01-21T19:46:28.310 INFO:teuthology.orchestra.run.smithi173.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 9.3 MB/s | 4.8 MB 00:00 2024-01-21T19:46:28.359 INFO:teuthology.orchestra.run.smithi173.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 172 kB/s | 49 kB 00:00 2024-01-21T19:46:28.393 INFO:teuthology.orchestra.run.smithi173.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 968 kB/s | 355 kB 00:00 2024-01-21T19:46:28.393 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:46:28.393 INFO:teuthology.orchestra.run.smithi173.stdout:Total 5.2 MB/s | 5.8 MB 00:01 2024-01-21T19:46:28.446 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-01-21T19:46:28.456 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-01-21T19:46:28.456 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-01-21T19:46:28.553 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-01-21T19:46:28.556 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-01-21T19:46:28.689 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-21T19:46:28.861 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-01-21T19:46:29.063 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-21T19:46:29.094 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-21T19:46:29.279 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-21T19:46:29.415 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-21T19:46:29.466 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:46:29.743 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:46:29.743 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-21T19:46:29.743 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-21T19:46:29.743 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-21T19:46:29.743 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-21T19:46:29.744 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-21T19:46:30.192 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-21T19:46:30.192 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:30.192 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-21T19:46:30.192 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-21T19:46:30.192 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-21T19:46:30.192 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-21T19:46:30.192 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:30.192 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:46:30.329 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-fuse 2024-01-21T19:46:30.359 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-21T19:46:30.732 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-21T19:46:30.949 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-21T19:46:31.060 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:58 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:31.194 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:46:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:46:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-21T19:46:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-21T19:46:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-21T19:46:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-21T19:46:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-21T19:46:31.901 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-21T19:46:31.901 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:31.901 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-01-21T19:46:31.901 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-21T19:46:31.901 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-21T19:46:31.901 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-21T19:46:31.902 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:31.902 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:46:32.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-fuse 2024-01-21T19:46:32.553 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:46:32.554 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:46:32.554 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-01-21T19:46:32.554 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:46:32.554 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-21T19:46:32.554 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-21T19:46:32.554 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-01-21T19:46:32.554 INFO:teuthology.orchestra.run.smithi169.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-21T19:46:32.554 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:32.554 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-21T19:46:32.554 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:46:32.554 INFO:teuthology.orchestra.run.smithi169.stdout:Install 2 Packages 2024-01-21T19:46:32.554 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:32.554 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 968 k 2024-01-21T19:46:32.555 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 3.2 M 2024-01-21T19:46:32.555 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-21T19:46:32.788 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:59 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:32.795 INFO:teuthology.orchestra.run.smithi169.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 15 MB/s | 885 kB 00:00 2024-01-21T19:46:32.936 INFO:teuthology.orchestra.run.smithi169.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 418 kB/s | 83 kB 00:00 2024-01-21T19:46:32.937 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:46:32.937 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.5 MB/s | 968 kB 00:00 2024-01-21T19:46:32.947 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-21T19:46:32.967 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-21T19:46:32.968 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-21T19:46:33.056 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-21T19:46:33.059 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-21T19:46:33.238 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-21T19:46:33.532 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-21T19:46:33.732 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T19:46:34.130 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T19:46:34.130 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-21T19:46:34.339 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:46:34.340 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:46:34.340 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-01-21T19:46:34.340 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:46:34.340 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-01-21T19:46:34.340 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-21T19:46:34.340 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-01-21T19:46:34.340 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-21T19:46:34.340 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:34.340 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-01-21T19:46:34.340 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:46:34.340 INFO:teuthology.orchestra.run.smithi173.stdout:Install 2 Packages 2024-01-21T19:46:34.341 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:34.341 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 968 k 2024-01-21T19:46:34.341 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 3.2 M 2024-01-21T19:46:34.341 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-01-21T19:46:34.485 INFO:teuthology.orchestra.run.smithi173.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 15 MB/s | 885 kB 00:00 2024-01-21T19:46:34.539 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-21T19:46:34.539 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:34.539 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-21T19:46:34.539 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-21T19:46:34.539 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:34.540 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:46:34.636 INFO:teuthology.orchestra.run.smithi173.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 401 kB/s | 83 kB 00:00 2024-01-21T19:46:34.636 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:46:34.636 INFO:teuthology.orchestra.run.smithi173.stdout:Total 3.2 MB/s | 968 kB 00:00 2024-01-21T19:46:34.646 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-01-21T19:46:34.667 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-01-21T19:46:34.668 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-01-21T19:46:34.750 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados-devel 2024-01-21T19:46:34.754 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-01-21T19:46:34.758 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-01-21T19:46:34.966 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-01-21T19:46:35.236 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-21T19:46:35.436 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T19:46:35.486 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:02 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:35.814 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T19:46:35.814 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-21T19:46:36.195 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-21T19:46:36.195 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:36.195 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-01-21T19:46:36.195 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-21T19:46:36.195 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:36.195 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:46:36.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados-devel 2024-01-21T19:46:36.955 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:46:36.956 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:46:36.956 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-01-21T19:46:36.956 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:46:36.956 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-21T19:46:36.956 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-21T19:46:36.956 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:36.956 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-21T19:46:36.957 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:46:36.957 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-01-21T19:46:36.957 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:36.957 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 131 k 2024-01-21T19:46:36.957 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 439 k 2024-01-21T19:46:36.957 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-21T19:46:37.008 INFO:teuthology.orchestra.run.smithi169.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.5 MB/s | 131 kB 00:00 2024-01-21T19:46:37.009 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:46:37.009 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.5 MB/s | 131 kB 00:00 2024-01-21T19:46:37.009 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-21T19:46:37.014 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-21T19:46:37.014 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-21T19:46:37.063 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-21T19:46:37.066 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-21T19:46:37.187 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-21T19:46:37.379 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:04 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:37.414 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:46:37.694 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:46:38.063 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:46:38.063 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:38.063 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-21T19:46:38.063 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-21T19:46:38.063 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:38.063 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:46:38.287 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs2 2024-01-21T19:46:38.861 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:46:38.862 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:46:38.862 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-01-21T19:46:38.862 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:46:38.862 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-01-21T19:46:38.863 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-21T19:46:38.863 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:38.863 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-01-21T19:46:38.863 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:46:38.863 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-01-21T19:46:38.863 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:38.863 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 131 k 2024-01-21T19:46:38.863 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 439 k 2024-01-21T19:46:38.863 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-01-21T19:46:38.916 INFO:teuthology.orchestra.run.smithi173.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 131 kB 00:00 2024-01-21T19:46:38.917 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:46:38.917 INFO:teuthology.orchestra.run.smithi173.stdout:Total 2.4 MB/s | 131 kB 00:00 2024-01-21T19:46:38.917 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-01-21T19:46:38.922 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-01-21T19:46:38.922 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-01-21T19:46:38.974 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-01-21T19:46:38.977 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-01-21T19:46:39.024 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:06 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:39.109 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-01-21T19:46:39.342 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:46:39.647 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:46:40.027 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:46:40.028 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:40.028 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-01-21T19:46:40.028 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-21T19:46:40.028 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:40.028 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:46:40.270 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs2 2024-01-21T19:46:40.442 INFO:teuthology.orchestra.run.smithi169.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:46:40.488 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:46:40.489 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-01-21T19:46:40.489 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:46:40.586 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs-devel 2024-01-21T19:46:41.004 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:08 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:41.315 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:08 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:42.433 INFO:teuthology.orchestra.run.smithi173.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:46:42.479 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:46:42.480 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-01-21T19:46:42.480 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:46:42.815 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:46:42.816 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:46:42.816 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-01-21T19:46:42.816 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:46:42.816 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-21T19:46:42.816 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-21T19:46:42.816 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:42.816 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-21T19:46:42.816 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:46:42.816 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-01-21T19:46:42.816 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:42.817 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 24 k 2024-01-21T19:46:42.817 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 94 k 2024-01-21T19:46:42.817 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-21T19:46:42.866 INFO:teuthology.orchestra.run.smithi169.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 497 kB/s | 24 kB 00:00 2024-01-21T19:46:42.867 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:46:42.867 INFO:teuthology.orchestra.run.smithi169.stdout:Total 484 kB/s | 24 kB 00:00 2024-01-21T19:46:42.868 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-21T19:46:42.870 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-21T19:46:42.870 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-21T19:46:42.872 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-21T19:46:42.876 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-21T19:46:42.929 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-21T19:46:42.979 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs-devel 2024-01-21T19:46:43.138 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:46:43.405 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:46:43.708 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:10 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:43.764 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:46:43.764 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:43.765 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-21T19:46:43.765 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-21T19:46:43.765 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:43.765 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:46:43.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados2 2024-01-21T19:46:44.612 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:11 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:45.180 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:46:45.181 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:46:45.181 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-01-21T19:46:45.181 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:46:45.181 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-01-21T19:46:45.181 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-21T19:46:45.181 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:45.181 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-01-21T19:46:45.182 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:46:45.182 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-01-21T19:46:45.182 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:45.182 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 24 k 2024-01-21T19:46:45.182 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 94 k 2024-01-21T19:46:45.182 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-01-21T19:46:45.232 INFO:teuthology.orchestra.run.smithi173.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 487 kB/s | 24 kB 00:00 2024-01-21T19:46:45.232 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:46:45.232 INFO:teuthology.orchestra.run.smithi173.stdout:Total 475 kB/s | 24 kB 00:00 2024-01-21T19:46:45.233 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-01-21T19:46:45.236 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-01-21T19:46:45.236 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-01-21T19:46:45.238 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-01-21T19:46:45.241 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-01-21T19:46:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-01-21T19:46:45.474 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:46:45.741 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:46:46.058 INFO:teuthology.orchestra.run.smithi169.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:46:46.083 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:46:46.083 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:46.083 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-01-21T19:46:46.084 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-21T19:46:46.084 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:46:46.084 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:46:46.104 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:46:46.105 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-01-21T19:46:46.105 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:46:46.198 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librbd1 2024-01-21T19:46:46.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados2 2024-01-21T19:46:46.924 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:13 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:46.952 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:13 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:48.360 INFO:teuthology.orchestra.run.smithi169.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:46:48.388 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:46:48.406 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:46:48.407 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-01-21T19:46:48.407 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:46:48.433 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:46:48.434 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-01-21T19:46:48.435 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:46:48.503 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rados 2024-01-21T19:46:48.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librbd1 2024-01-21T19:46:49.235 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:16 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:49.619 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:16 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:50.660 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:46:50.705 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:46:50.707 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-01-21T19:46:50.707 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:46:50.806 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rgw 2024-01-21T19:46:51.066 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:46:51.112 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:46:51.113 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-01-21T19:46:51.113 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:46:51.221 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rados 2024-01-21T19:46:51.537 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:18 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:51.955 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:18 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:52.972 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:46:53.018 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:46:53.019 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-01-21T19:46:53.019 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:46:53.121 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-cephfs 2024-01-21T19:46:53.385 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:46:53.430 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:46:53.431 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-01-21T19:46:53.431 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:46:53.546 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rgw 2024-01-21T19:46:53.851 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:20 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:54.281 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:21 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:55.296 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:46:55.341 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:46:55.342 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-01-21T19:46:55.343 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:46:55.444 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rbd 2024-01-21T19:46:55.715 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:46:55.763 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:46:55.764 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-01-21T19:46:55.764 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:46:55.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-cephfs 2024-01-21T19:46:56.171 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:23 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:56.602 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:23 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:57.588 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:46:57.634 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:46:57.635 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-01-21T19:46:57.635 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:46:57.739 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-fuse 2024-01-21T19:46:58.035 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:46:58.080 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:46:58.081 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-01-21T19:46:58.081 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:46:58.191 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rbd 2024-01-21T19:46:58.467 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:25 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:58.926 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:25 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:46:59.959 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:46:59.959 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:46:59.959 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-01-21T19:46:59.959 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:46:59.959 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-21T19:46:59.959 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-21T19:46:59.959 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:59.959 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-21T19:46:59.960 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:46:59.960 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-01-21T19:46:59.960 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:46:59.960 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 89 k 2024-01-21T19:46:59.960 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 298 k 2024-01-21T19:46:59.960 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-21T19:47:00.014 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.6 MB/s | 89 kB 00:00 2024-01-21T19:47:00.015 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:47:00.015 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.6 MB/s | 89 kB 00:00 2024-01-21T19:47:00.015 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-21T19:47:00.024 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-21T19:47:00.024 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-21T19:47:00.075 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-21T19:47:00.078 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-21T19:47:00.189 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-21T19:47:00.374 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:47:00.420 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:47:00.421 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-01-21T19:47:00.421 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:47:00.435 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:47:00.528 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-fuse 2024-01-21T19:47:00.729 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:47:01.093 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:47:01.093 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:47:01.093 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-21T19:47:01.093 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-21T19:47:01.093 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:47:01.093 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:47:01.253 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:28 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:47:01.330 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-mirror 2024-01-21T19:47:02.071 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:29 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:47:02.737 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:47:02.737 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:47:02.737 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-01-21T19:47:02.737 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:47:02.737 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-01-21T19:47:02.737 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-21T19:47:02.738 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:47:02.738 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-01-21T19:47:02.738 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:47:02.738 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-01-21T19:47:02.738 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:47:02.738 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 89 k 2024-01-21T19:47:02.739 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 298 k 2024-01-21T19:47:02.739 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-01-21T19:47:02.781 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.1 MB/s | 89 kB 00:00 2024-01-21T19:47:02.781 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:47:02.781 INFO:teuthology.orchestra.run.smithi173.stdout:Total 2.0 MB/s | 89 kB 00:00 2024-01-21T19:47:02.781 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-01-21T19:47:02.791 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-01-21T19:47:02.791 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-01-21T19:47:02.840 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-01-21T19:47:02.843 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-01-21T19:47:02.941 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-01-21T19:47:03.164 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:47:03.455 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:47:03.532 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:47:03.533 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:47:03.533 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-01-21T19:47:03.533 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:47:03.533 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-21T19:47:03.533 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T19:47:03.533 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:47:03.533 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-21T19:47:03.533 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:47:03.533 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-01-21T19:47:03.534 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:47:03.534 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 3.7 M 2024-01-21T19:47:03.534 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 18 M 2024-01-21T19:47:03.534 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-21T19:47:03.617 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 45 MB/s | 3.7 MB 00:00 2024-01-21T19:47:03.617 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:47:03.618 INFO:teuthology.orchestra.run.smithi169.stdout:Total 44 MB/s | 3.7 MB 00:00 2024-01-21T19:47:03.618 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-21T19:47:03.630 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-21T19:47:03.630 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-21T19:47:03.692 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-21T19:47:03.695 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-21T19:47:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:47:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:47:03.809 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-01-21T19:47:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-21T19:47:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:47:03.810 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:47:04.057 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-mirror 2024-01-21T19:47:04.162 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-21T19:47:04.469 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:47:04.798 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:31 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:47:04.879 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:47:05.260 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:47:05.260 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:47:05.260 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-21T19:47:05.260 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-21T19:47:05.260 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:47:05.260 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:47:05.502 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-nbd 2024-01-21T19:47:06.242 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:33 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:47:06.294 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:47:06.295 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:47:06.295 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-01-21T19:47:06.295 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:47:06.295 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-01-21T19:47:06.295 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T19:47:06.295 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:47:06.295 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-01-21T19:47:06.295 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:47:06.295 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-01-21T19:47:06.295 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:47:06.295 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 3.7 M 2024-01-21T19:47:06.295 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 18 M 2024-01-21T19:47:06.295 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-01-21T19:47:06.375 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 47 MB/s | 3.7 MB 00:00 2024-01-21T19:47:06.376 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:47:06.376 INFO:teuthology.orchestra.run.smithi173.stdout:Total 46 MB/s | 3.7 MB 00:00 2024-01-21T19:47:06.376 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-01-21T19:47:06.388 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-01-21T19:47:06.388 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-01-21T19:47:06.451 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-01-21T19:47:06.454 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-01-21T19:47:06.917 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-01-21T19:47:07.172 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:47:07.590 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:47:07.715 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-21T19:47:07.716 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:47:07.716 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-01-21T19:47:07.716 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:47:07.716 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-21T19:47:07.716 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-21T19:47:07.716 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:47:07.716 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-21T19:47:07.716 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-21T19:47:07.717 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-01-21T19:47:07.717 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:47:07.717 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 173 k 2024-01-21T19:47:07.717 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 649 k 2024-01-21T19:47:07.717 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-21T19:47:07.762 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.8 MB/s | 173 kB 00:00 2024-01-21T19:47:07.762 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:47:07.762 INFO:teuthology.orchestra.run.smithi169.stdout:Total 3.7 MB/s | 173 kB 00:00 2024-01-21T19:47:07.763 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-21T19:47:07.773 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-21T19:47:07.774 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-21T19:47:07.823 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-21T19:47:07.826 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-21T19:47:07.946 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-21T19:47:07.964 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:47:07.964 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:47:07.964 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-01-21T19:47:07.964 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-21T19:47:07.964 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:47:07.964 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:47:08.142 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:47:08.199 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-nbd 2024-01-21T19:47:08.421 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:47:08.802 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:47:08.802 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:47:08.802 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-21T19:47:08.802 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-21T19:47:08.802 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-21T19:47:08.802 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-21T19:47:08.944 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:35 ago on Sun 21 Jan 2024 07:44:33 PM UTC. 2024-01-21T19:47:09.009 DEBUG:teuthology.parallel:result is None 2024-01-21T19:47:10.425 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-01-21T19:47:10.425 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:47:10.425 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-01-21T19:47:10.426 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:47:10.426 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-01-21T19:47:10.426 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-21T19:47:10.426 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:47:10.426 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-01-21T19:47:10.426 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-01-21T19:47:10.426 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-01-21T19:47:10.427 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:47:10.427 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 173 k 2024-01-21T19:47:10.427 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 649 k 2024-01-21T19:47:10.428 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-01-21T19:47:10.478 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.3 MB/s | 173 kB 00:00 2024-01-21T19:47:10.479 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:47:10.479 INFO:teuthology.orchestra.run.smithi173.stdout:Total 3.2 MB/s | 173 kB 00:00 2024-01-21T19:47:10.479 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-01-21T19:47:10.490 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-01-21T19:47:10.490 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-01-21T19:47:10.540 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-01-21T19:47:10.542 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-01-21T19:47:10.659 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-01-21T19:47:10.846 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:47:11.168 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:47:11.531 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:47:11.532 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:47:11.532 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-01-21T19:47:11.532 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-21T19:47:11.532 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:47:11.532 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-01-21T19:47:11.736 DEBUG:teuthology.parallel:result is None 2024-01-21T19:47:11.737 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T19:47:11.737 INFO:teuthology.packaging:ref: None 2024-01-21T19:47:11.737 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T19:47:11.737 INFO:teuthology.packaging:branch: None 2024-01-21T19:47:11.737 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T19:47:11.737 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T19:47:11.886 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-21T19:47:11.914 INFO:teuthology.orchestra.run.smithi169.stdout:16.2.4-0.el8 2024-01-21T19:47:11.914 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-21T19:47:11.914 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-21T19:47:11.916 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T19:47:11.916 INFO:teuthology.packaging:ref: None 2024-01-21T19:47:11.916 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T19:47:11.916 INFO:teuthology.packaging:branch: None 2024-01-21T19:47:11.916 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T19:47:11.916 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T19:47:12.053 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-21T19:47:12.075 INFO:teuthology.orchestra.run.smithi173.stdout:16.2.4-0.el8 2024-01-21T19:47:12.075 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-21T19:47:12.075 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-21T19:47:12.077 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-21T19:47:12.077 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-01-21T19:47:12.077 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-21T19:47:12.110 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-01-21T19:47:12.110 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-21T19:47:12.144 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-21T19:47:12.144 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-01-21T19:47:12.144 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/daemon-helper 2024-01-21T19:47:12.182 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-21T19:47:12.252 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-01-21T19:47:12.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-01-21T19:47:12.279 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-21T19:47:12.346 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-21T19:47:12.347 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-01-21T19:47:12.347 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-21T19:47:12.376 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-21T19:47:12.444 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-01-21T19:47:12.445 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-21T19:47:12.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-21T19:47:12.539 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-21T19:47:12.540 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-01-21T19:47:12.540 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/stdin-killer 2024-01-21T19:47:12.567 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-21T19:47:12.634 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-01-21T19:47:12.634 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-01-21T19:47:12.661 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-21T19:47:12.728 INFO:teuthology.run_tasks:Running task print... 2024-01-21T19:47:12.737 INFO:teuthology.task.print:**** done install task... 2024-01-21T19:47:12.737 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-21T19:47:12.840 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec'} 2024-01-21T19:47:12.840 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-01-21T19:47:12.841 INFO:tasks.cephadm:Cluster fsid is df2470f2-b895-11ee-95b0-87774f69a715 2024-01-21T19:47:12.841 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-21T19:47:12.841 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-01-21T19:47:12.841 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi169': '172.21.15.169', 'mon.smithi173': '172.21.15.173'} 2024-01-21T19:47:12.841 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-21T19:47:12.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo hostname $(hostname -s) 2024-01-21T19:47:12.870 DEBUG:teuthology.orchestra.run.smithi173:> sudo hostname $(hostname -s) 2024-01-21T19:47:12.897 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-01-21T19:47:12.897 DEBUG:teuthology.orchestra.run.smithi169:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-21T19:47:13.044 INFO:teuthology.orchestra.run.smithi169.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 21 19:47 /home/ubuntu/cephtest/cephadm 2024-01-21T19:47:13.045 DEBUG:teuthology.orchestra.run.smithi173:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-21T19:47:13.179 INFO:teuthology.orchestra.run.smithi173.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 21 19:47 /home/ubuntu/cephtest/cephadm 2024-01-21T19:47:13.180 DEBUG:teuthology.orchestra.run.smithi169:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-21T19:47:13.199 DEBUG:teuthology.orchestra.run.smithi173:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-21T19:47:13.227 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph 2024-01-21T19:47:13.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph 2024-01-21T19:47:13.293 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 777 /etc/ceph 2024-01-21T19:47:13.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /etc/ceph 2024-01-21T19:47:13.361 INFO:tasks.cephadm:Writing seed config... 2024-01-21T19:47:13.361 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-01-21T19:47:13.362 INFO:tasks.cephadm: override: [client] debug client = 20 2024-01-21T19:47:13.362 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-01-21T19:47:13.362 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-01-21T19:47:13.362 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-01-21T19:47:13.362 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-01-21T19:47:13.362 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-01-21T19:47:13.362 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-01-21T19:47:13.362 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-01-21T19:47:13.362 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-01-21T19:47:13.362 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-01-21T19:47:13.362 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-01-21T19:47:13.362 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-01-21T19:47:13.362 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-01-21T19:47:13.362 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-01-21T19:47:13.363 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-01-21T19:47:13.363 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-21T19:47:13.363 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-21T19:47:13.363 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-21T19:47:13.363 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-21T19:47:13.363 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-21T19:47:13.363 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-01-21T19:47:13.363 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-01-21T19:47:13.363 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-01-21T19:47:13.363 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-01-21T19:47:13.363 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-01-21T19:47:13.363 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-01-21T19:47:13.363 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-01-21T19:47:13.363 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-01-21T19:47:13.363 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-21T19:47:13.363 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-21T19:47:13.363 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-01-21T19:47:13.364 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-01-21T19:47:13.364 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-01-21T19:47:13.364 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-01-21T19:47:13.364 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-01-21T19:47:13.364 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-01-21T19:47:13.364 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-01-21T19:47:13.365 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-01-21T19:47:13.365 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-21T19:47:13.392 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = df2470f2-b895-11ee-95b0-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-01-21T19:47:13.392 DEBUG:teuthology.orchestra.run.smithi169:mon.smithi169> sudo journalctl -f -n 0 -u ceph-df2470f2-b895-11ee-95b0-87774f69a715@mon.smithi169.service 2024-01-21T19:47:13.434 INFO:tasks.cephadm:Bootstrapping... 2024-01-21T19:47:13.434 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid df2470f2-b895-11ee-95b0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.169 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-21T19:47:13.460 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:-- Logs begin at Sun 2024-01-21 19:37:20 UTC. -- 2024-01-21T19:47:13.693 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-21T19:47:14.486 INFO:teuthology.orchestra.run.smithi169.stderr:/bin/podman: 4.6.1 2024-01-21T19:47:14.494 INFO:teuthology.orchestra.run.smithi169.stderr:Verifying podman|docker is present... 2024-01-21T19:47:14.494 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-21T19:47:14.517 INFO:teuthology.orchestra.run.smithi169.stderr:/bin/podman: 4.6.1 2024-01-21T19:47:14.526 INFO:teuthology.orchestra.run.smithi169.stderr:Verifying lvm2 is present... 2024-01-21T19:47:14.527 INFO:teuthology.orchestra.run.smithi169.stderr:Verifying time synchronization is in place... 2024-01-21T19:47:14.527 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl is-enabled chrony.service 2024-01-21T19:47:14.534 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-21T19:47:14.535 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl is-active chrony.service 2024-01-21T19:47:14.540 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: inactive 2024-01-21T19:47:14.541 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-21T19:47:14.547 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: enabled 2024-01-21T19:47:14.548 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl is-active chronyd.service 2024-01-21T19:47:14.553 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: active 2024-01-21T19:47:14.554 INFO:teuthology.orchestra.run.smithi169.stderr:Unit chronyd.service is enabled and running 2024-01-21T19:47:14.554 INFO:teuthology.orchestra.run.smithi169.stderr:Repeating the final host check... 2024-01-21T19:47:14.554 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-21T19:47:14.575 INFO:teuthology.orchestra.run.smithi169.stderr:/bin/podman: 4.6.1 2024-01-21T19:47:14.590 INFO:teuthology.orchestra.run.smithi169.stderr:podman|docker (/bin/podman) is present 2024-01-21T19:47:14.590 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl is present 2024-01-21T19:47:14.590 INFO:teuthology.orchestra.run.smithi169.stderr:lvcreate is present 2024-01-21T19:47:14.590 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl is-enabled chrony.service 2024-01-21T19:47:14.596 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-21T19:47:14.596 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl is-active chrony.service 2024-01-21T19:47:14.602 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: inactive 2024-01-21T19:47:14.602 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-21T19:47:14.608 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: enabled 2024-01-21T19:47:14.609 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl is-active chronyd.service 2024-01-21T19:47:14.614 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: active 2024-01-21T19:47:14.615 INFO:teuthology.orchestra.run.smithi169.stderr:Unit chronyd.service is enabled and running 2024-01-21T19:47:14.615 INFO:teuthology.orchestra.run.smithi169.stderr:Host looks OK 2024-01-21T19:47:14.615 INFO:teuthology.orchestra.run.smithi169.stderr:Cluster fsid: df2470f2-b895-11ee-95b0-87774f69a715 2024-01-21T19:47:14.615 INFO:teuthology.orchestra.run.smithi169.stderr:Acquiring lock 140394404059624 on /run/cephadm/df2470f2-b895-11ee-95b0-87774f69a715.lock 2024-01-21T19:47:14.615 INFO:teuthology.orchestra.run.smithi169.stderr:Lock 140394404059624 acquired on /run/cephadm/df2470f2-b895-11ee-95b0-87774f69a715.lock 2024-01-21T19:47:14.616 INFO:teuthology.orchestra.run.smithi169.stderr:Verifying IP 172.21.15.169 port 3300 ... 2024-01-21T19:47:14.616 INFO:teuthology.orchestra.run.smithi169.stderr:Verifying IP 172.21.15.169 port 6789 ... 2024-01-21T19:47:14.616 INFO:teuthology.orchestra.run.smithi169.stderr:Base mon IP is 172.21.15.169, final addrv is [v2:172.21.15.169:3300,v1:172.21.15.169:6789] 2024-01-21T19:47:14.616 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /sbin/ip route ls 2024-01-21T19:47:14.618 INFO:teuthology.orchestra.run.smithi169.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.169 metric 100 2024-01-21T19:47:14.618 INFO:teuthology.orchestra.run.smithi169.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.169 metric 100 2024-01-21T19:47:14.619 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /sbin/ip -6 route ls 2024-01-21T19:47:14.621 INFO:teuthology.orchestra.run.smithi169.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-01-21T19:47:14.621 INFO:teuthology.orchestra.run.smithi169.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-01-21T19:47:14.621 INFO:teuthology.orchestra.run.smithi169.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2024-01-21T19:47:14.622 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /sbin/ip -6 addr ls 2024-01-21T19:47:14.624 INFO:teuthology.orchestra.run.smithi169.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-21T19:47:14.624 INFO:teuthology.orchestra.run.smithi169.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-01-21T19:47:14.624 INFO:teuthology.orchestra.run.smithi169.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-21T19:47:14.624 INFO:teuthology.orchestra.run.smithi169.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-01-21T19:47:14.625 INFO:teuthology.orchestra.run.smithi169.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe88:7f4d/64 scope link 2024-01-21T19:47:14.625 INFO:teuthology.orchestra.run.smithi169.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-21T19:47:14.625 INFO:teuthology.orchestra.run.smithi169.stderr:Mon IP 172.21.15.169 is in CIDR network 172.21.0.0/20 2024-01-21T19:47:14.625 INFO:teuthology.orchestra.run.smithi169.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-21T19:47:14.626 INFO:teuthology.orchestra.run.smithi169.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-01-21T19:47:14.626 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-01-21T19:47:14.760 INFO:teuthology.orchestra.run.smithi169.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-01-21T19:47:21.353 INFO:teuthology.orchestra.run.smithi169.stderr:/bin/podman: Getting image source signatures 2024-01-21T19:47:21.362 INFO:teuthology.orchestra.run.smithi169.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-01-21T19:47:21.362 INFO:teuthology.orchestra.run.smithi169.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-01-21T19:47:33.533 INFO:teuthology.orchestra.run.smithi169.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-21T19:47:33.599 INFO:teuthology.orchestra.run.smithi169.stderr:/bin/podman: Writing manifest to image destination 2024-01-21T19:47:33.843 INFO:teuthology.orchestra.run.smithi169.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-21T19:47:33.855 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-01-21T19:47:34.386 INFO:teuthology.orchestra.run.smithi169.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-21T19:47:34.740 INFO:teuthology.orchestra.run.smithi169.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-21T19:47:34.740 INFO:teuthology.orchestra.run.smithi169.stderr:Extracting ceph user uid/gid from container image... 2024-01-21T19:47:34.740 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-01-21T19:47:35.016 INFO:teuthology.orchestra.run.smithi169.stderr:stat: 167 167 2024-01-21T19:47:35.333 INFO:teuthology.orchestra.run.smithi169.stderr:Creating initial keys... 2024-01-21T19:47:35.333 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-21T19:47:35.615 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-authtool: AQDXdK1lgosgJBAA896KDO9DE84HWAAYQmj1sg== 2024-01-21T19:47:35.952 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-21T19:47:36.232 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-authtool: AQDYdK1lWMNzDRAAL2HmEv1Vstlw3dPxjpOugw== 2024-01-21T19:47:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-21T19:47:36.832 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-authtool: AQDYdK1lmkCFMBAAe7YZApehu98xsp5Zgx3aHQ== 2024-01-21T19:47:37.151 INFO:teuthology.orchestra.run.smithi169.stderr:Creating initial monmap... 2024-01-21T19:47:37.151 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpgw36ctkm:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid df2470f2-b895-11ee-95b0-87774f69a715 --addv smithi169 [v2:172.21.15.169:3300,v1:172.21.15.169:6789] /tmp/monmap 2024-01-21T19:47:37.449 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-21T19:47:37.449 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to df2470f2-b895-11ee-95b0-87774f69a715 2024-01-21T19:47:37.449 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-21T19:47:37.790 INFO:teuthology.orchestra.run.smithi169.stderr:monmaptool for smithi169 [v2:172.21.15.169:3300,v1:172.21.15.169:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-21T19:47:37.790 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/monmaptool: set fsid to df2470f2-b895-11ee-95b0-87774f69a715 2024-01-21T19:47:37.790 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-21T19:47:37.790 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-01-21T19:47:37.790 INFO:teuthology.orchestra.run.smithi169.stderr:Creating mon... 2024-01-21T19:47:37.791 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/mon.smithi169:/var/lib/ceph/mon/ceph-smithi169:z -v /tmp/ceph-tmpg1wjwcif:/tmp/keyring:z -v /tmp/ceph-tmpgw36ctkm:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi169 --fsid df2470f2-b895-11ee-95b0-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-01-21T19:47:38.097 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.062+0000 7fc17b1f2700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-21T19:47:38.097 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 1 imported monmap: 2024-01-21T19:47:38.097 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: epoch 0 2024-01-21T19:47:38.098 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: fsid df2470f2-b895-11ee-95b0-87774f69a715 2024-01-21T19:47:38.098 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: last_changed 2024-01-21T19:47:37.438649+0000 2024-01-21T19:47:38.098 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: created 2024-01-21T19:47:37.438649+0000 2024-01-21T19:47:38.098 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-01-21T19:47:38.098 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-01-21T19:47:38.098 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon.smithi169 2024-01-21T19:47:38.098 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: 2024-01-21T19:47:38.098 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 0 /usr/bin/ceph-mon: set fsid to df2470f2-b895-11ee-95b0-87774f69a715 2024-01-21T19:47:38.098 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: RocksDB version: 6.8.1 2024-01-21T19:47:38.098 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: 2024-01-21T19:47:38.098 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-21T19:47:38.098 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Compile date May 13 2021 2024-01-21T19:47:38.098 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: DB SUMMARY 2024-01-21T19:47:38.098 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: 2024-01-21T19:47:38.098 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi169/store.db dir, Total Num: 0, files: 2024-01-21T19:47:38.099 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: 2024-01-21T19:47:38.099 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi169/store.db: 2024-01-21T19:47:38.099 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: 2024-01-21T19:47:38.099 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.error_if_exists: 0 2024-01-21T19:47:38.099 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.create_if_missing: 1 2024-01-21T19:47:38.099 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.paranoid_checks: 1 2024-01-21T19:47:38.099 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.env: 0x562cd6b971c0 2024-01-21T19:47:38.099 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.fs: Posix File System 2024-01-21T19:47:38.099 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.info_log: 0x562cd8e62ec0 2024-01-21T19:47:38.099 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-21T19:47:38.099 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.statistics: (nil) 2024-01-21T19:47:38.099 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.use_fsync: 0 2024-01-21T19:47:38.099 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.max_log_file_size: 0 2024-01-21T19:47:38.099 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-21T19:47:38.100 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-21T19:47:38.100 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-21T19:47:38.100 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-21T19:47:38.100 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.allow_fallocate: 1 2024-01-21T19:47:38.100 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-21T19:47:38.100 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-21T19:47:38.100 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.use_direct_reads: 0 2024-01-21T19:47:38.100 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-21T19:47:38.100 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-21T19:47:38.101 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.db_log_dir: 2024-01-21T19:47:38.101 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi169/store.db 2024-01-21T19:47:38.101 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-21T19:47:38.101 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.max_subcompactions: 1 2024-01-21T19:47:38.101 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.max_background_flushes: -1 2024-01-21T19:47:38.101 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-21T19:47:38.101 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-21T19:47:38.101 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-21T19:47:38.101 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-21T19:47:38.101 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-21T19:47:38.101 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-21T19:47:38.101 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-21T19:47:38.101 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.write_buffer_manager: 0x562cd8e69530 2024-01-21T19:47:38.101 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-21T19:47:38.101 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-21T19:47:38.102 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-21T19:47:38.102 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-21T19:47:38.102 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-21T19:47:38.102 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-21T19:47:38.102 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-21T19:47:38.102 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-21T19:47:38.102 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-21T19:47:38.102 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.unordered_write: 0 2024-01-21T19:47:38.102 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-21T19:47:38.102 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-21T19:47:38.102 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-21T19:47:38.102 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-21T19:47:38.102 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.row_cache: None 2024-01-21T19:47:38.103 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.wal_filter: None 2024-01-21T19:47:38.103 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-21T19:47:38.103 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-21T19:47:38.103 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.preserve_deletes: 0 2024-01-21T19:47:38.103 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.two_write_queues: 0 2024-01-21T19:47:38.103 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-21T19:47:38.104 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.atomic_flush: 0 2024-01-21T19:47:38.104 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-21T19:47:38.104 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-21T19:47:38.104 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-21T19:47:38.104 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.log_readahead_size: 0 2024-01-21T19:47:38.104 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-21T19:47:38.105 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.max_background_jobs: 2 2024-01-21T19:47:38.105 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.max_background_compactions: -1 2024-01-21T19:47:38.105 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-21T19:47:38.105 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-21T19:47:38.105 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-21T19:47:38.105 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-21T19:47:38.105 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-21T19:47:38.106 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-21T19:47:38.106 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-21T19:47:38.106 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-21T19:47:38.106 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.max_open_files: -1 2024-01-21T19:47:38.106 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-21T19:47:38.106 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-21T19:47:38.106 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-21T19:47:38.107 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-21T19:47:38.107 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Compression algorithms supported: 2024-01-21T19:47:38.107 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-21T19:47:38.107 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: kZSTD supported: 0 2024-01-21T19:47:38.107 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: kXpressCompression supported: 0 2024-01-21T19:47:38.107 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-21T19:47:38.107 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: kLZ4Compression supported: 1 2024-01-21T19:47:38.108 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: kBZip2Compression supported: 0 2024-01-21T19:47:38.108 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: kZlibCompression supported: 1 2024-01-21T19:47:38.108 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: kSnappyCompression supported: 1 2024-01-21T19:47:38.108 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.063+0000 7fc17b1f2700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-21T19:47:38.108 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-01-21T19:47:38.108 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: 2024-01-21T19:47:38.109 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi169/store.db/MANIFEST-000001 2024-01-21T19:47:38.109 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: 2024-01-21T19:47:38.109 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-21T19:47:38.109 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: 2024-01-21T19:47:38.109 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-21T19:47:38.109 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.merge_operator: 2024-01-21T19:47:38.109 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.compaction_filter: None 2024-01-21T19:47:38.110 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-21T19:47:38.110 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-21T19:47:38.110 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-21T19:47:38.110 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562cd8d7a120) 2024-01-21T19:47:38.110 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-01-21T19:47:38.110 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-21T19:47:38.110 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-21T19:47:38.111 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-01-21T19:47:38.111 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: index_type: 0 2024-01-21T19:47:38.111 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-01-21T19:47:38.111 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-01-21T19:47:38.111 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-01-21T19:47:38.111 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-01-21T19:47:38.112 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: checksum: 1 2024-01-21T19:47:38.112 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-01-21T19:47:38.112 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: block_cache: 0x562cd8db0d10 2024-01-21T19:47:38.112 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-01-21T19:47:38.112 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-01-21T19:47:38.112 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-01-21T19:47:38.112 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-01-21T19:47:38.112 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-01-21T19:47:38.113 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-01-21T19:47:38.113 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-01-21T19:47:38.113 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-01-21T19:47:38.113 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-01-21T19:47:38.113 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-01-21T19:47:38.113 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-01-21T19:47:38.113 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-01-21T19:47:38.114 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-01-21T19:47:38.114 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-01-21T19:47:38.114 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-01-21T19:47:38.114 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-21T19:47:38.114 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-01-21T19:47:38.114 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-01-21T19:47:38.114 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-01-21T19:47:38.115 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: format_version: 2 2024-01-21T19:47:38.115 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-01-21T19:47:38.115 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: block_align: 0 2024-01-21T19:47:38.115 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: 2024-01-21T19:47:38.115 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-21T19:47:38.115 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-21T19:47:38.115 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.compression: NoCompression 2024-01-21T19:47:38.116 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-21T19:47:38.116 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-21T19:47:38.116 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-21T19:47:38.116 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.num_levels: 7 2024-01-21T19:47:38.116 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-21T19:47:38.116 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-21T19:47:38.117 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-21T19:47:38.117 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-21T19:47:38.117 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-21T19:47:38.117 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-21T19:47:38.117 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-21T19:47:38.118 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-21T19:47:38.118 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-21T19:47:38.118 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-21T19:47:38.118 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-21T19:47:38.118 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-21T19:47:38.118 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-21T19:47:38.119 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-21T19:47:38.119 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-21T19:47:38.119 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-21T19:47:38.119 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-21T19:47:38.119 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-21T19:47:38.119 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-21T19:47:38.120 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-21T19:47:38.120 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-21T19:47:38.120 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-21T19:47:38.120 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-21T19:47:38.120 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-21T19:47:38.120 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-21T19:47:38.121 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-21T19:47:38.121 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-21T19:47:38.121 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-21T19:47:38.121 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-21T19:47:38.121 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-21T19:47:38.121 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-21T19:47:38.122 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-21T19:47:38.122 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-21T19:47:38.122 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-21T19:47:38.122 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-21T19:47:38.122 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-21T19:47:38.122 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-21T19:47:38.123 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-21T19:47:38.123 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-21T19:47:38.123 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-21T19:47:38.123 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-21T19:47:38.123 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-21T19:47:38.123 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-21T19:47:38.123 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-21T19:47:38.124 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-21T19:47:38.124 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-21T19:47:38.124 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-21T19:47:38.124 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.table_properties_collectors: 2024-01-21T19:47:38.124 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.inplace_update_support: 0 2024-01-21T19:47:38.125 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-21T19:47:38.125 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-21T19:47:38.125 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-21T19:47:38.125 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-21T19:47:38.125 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.bloom_locality: 0 2024-01-21T19:47:38.125 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.max_successive_merges: 0 2024-01-21T19:47:38.126 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-21T19:47:38.126 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-21T19:47:38.126 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-21T19:47:38.126 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-21T19:47:38.126 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.ttl: 2592000 2024-01-21T19:47:38.126 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.064+0000 7fc17b1f2700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-21T19:47:38.127 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.065+0000 7fc17b1f2700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi169/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-21T19:47:38.127 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: 2024-01-21T19:47:38.127 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.065+0000 7fc17b1f2700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-21T19:47:38.127 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: 2024-01-21T19:47:38.127 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.065+0000 7fc17b1f2700 4 rocksdb: DB pointer 0x562cd8e75800 2024-01-21T19:47:38.127 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.066+0000 7fc16408b700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-21T19:47:38.127 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.066+0000 7fc16408b700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-21T19:47:38.128 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-01-21T19:47:38.128 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T19:47:38.128 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-21T19:47:38.128 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-21T19:47:38.128 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T19:47:38.128 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-21T19:47:38.128 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-21T19:47:38.129 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T19:47:38.129 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: 2024-01-21T19:47:38.129 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T19:47:38.129 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:47:38.129 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:47:38.129 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T19:47:38.130 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T19:47:38.130 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: 2024-01-21T19:47:38.130 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T19:47:38.130 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:47:38.130 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:47:38.130 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T19:47:38.130 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T19:47:38.131 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T19:47:38.131 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T19:47:38.131 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T19:47:38.131 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-21T19:47:38.131 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:47:38.131 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:47:38.131 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T19:47:38.132 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: 2024-01-21T19:47:38.132 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-21T19:47:38.132 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: 2024-01-21T19:47:38.132 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T19:47:38.132 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:47:38.132 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:47:38.132 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T19:47:38.132 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T19:47:38.133 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: 2024-01-21T19:47:38.133 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T19:47:38.133 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:47:38.133 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:47:38.133 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T19:47:38.133 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T19:47:38.133 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T19:47:38.133 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T19:47:38.133 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T19:47:38.133 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-21T19:47:38.134 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:47:38.134 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:47:38.134 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T19:47:38.134 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: 2024-01-21T19:47:38.134 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-21T19:47:38.134 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: 2024-01-21T19:47:38.134 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.067+0000 7fc17b1f2700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-01-21T19:47:38.134 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.067+0000 7fc17b1f2700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-01-21T19:47:38.134 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:47:38.067+0000 7fc17b1f2700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi169 for mon.smithi169 2024-01-21T19:47:38.423 INFO:teuthology.orchestra.run.smithi169.stderr:create mon.smithi169 on 2024-01-21T19:47:38.423 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/df2470f2-b895-11ee-95b0-87774f69a715 2024-01-21T19:47:38.428 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl disable ceph.target 2024-01-21T19:47:38.434 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-01-21T19:47:38.517 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl enable ceph.target 2024-01-21T19:47:38.523 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-21T19:47:38.604 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl start ceph.target 2024-01-21T19:47:38.611 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl enable ceph-df2470f2-b895-11ee-95b0-87774f69a715.target 2024-01-21T19:47:38.617 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-df2470f2-b895-11ee-95b0-87774f69a715.target → /etc/systemd/system/ceph-df2470f2-b895-11ee-95b0-87774f69a715.target. 2024-01-21T19:47:38.617 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-df2470f2-b895-11ee-95b0-87774f69a715.target → /etc/systemd/system/ceph-df2470f2-b895-11ee-95b0-87774f69a715.target. 2024-01-21T19:47:38.697 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl start ceph-df2470f2-b895-11ee-95b0-87774f69a715.target 2024-01-21T19:47:38.705 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl daemon-reload 2024-01-21T19:47:38.791 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl stop ceph-df2470f2-b895-11ee-95b0-87774f69a715@mon.smithi169 2024-01-21T19:47:38.815 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl reset-failed ceph-df2470f2-b895-11ee-95b0-87774f69a715@mon.smithi169 2024-01-21T19:47:38.820 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: Failed to reset failed state of unit ceph-df2470f2-b895-11ee-95b0-87774f69a715@mon.smithi169.service: Unit ceph-df2470f2-b895-11ee-95b0-87774f69a715@mon.smithi169.service not loaded. 2024-01-21T19:47:38.821 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl enable ceph-df2470f2-b895-11ee-95b0-87774f69a715@mon.smithi169 2024-01-21T19:47:38.828 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: Created symlink /etc/systemd/system/ceph-df2470f2-b895-11ee-95b0-87774f69a715.target.wants/ceph-df2470f2-b895-11ee-95b0-87774f69a715@mon.smithi169.service → /etc/systemd/system/ceph-df2470f2-b895-11ee-95b0-87774f69a715@.service. 2024-01-21T19:47:38.915 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl start ceph-df2470f2-b895-11ee-95b0-87774f69a715@mon.smithi169 2024-01-21T19:47:39.220 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T19:47:39.229 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: disabled 2024-01-21T19:47:39.229 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T19:47:39.237 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: inactive 2024-01-21T19:47:39.238 INFO:teuthology.orchestra.run.smithi169.stderr:firewalld.service is not enabled 2024-01-21T19:47:39.238 INFO:teuthology.orchestra.run.smithi169.stderr:Not possible to enable service . firewalld.service is not available 2024-01-21T19:47:39.238 INFO:teuthology.orchestra.run.smithi169.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-21T19:47:39.239 INFO:teuthology.orchestra.run.smithi169.stderr:Waiting for mon to start... 2024-01-21T19:47:39.239 INFO:teuthology.orchestra.run.smithi169.stderr:Waiting for mon... 2024-01-21T19:47:39.239 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/mon.smithi169:/var/lib/ceph/mon/ceph-smithi169:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-01-21T19:47:39.654 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.659+0000 7f76c3ada700 1 Processor -- start 2024-01-21T19:47:39.654 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.659+0000 7f76c3ada700 1 -- start start 2024-01-21T19:47:39.654 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.659+0000 7f76c3ada700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76bc0703d0 0x7f76bc100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:39.654 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.660+0000 7f76c3ada700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc0fc730 con 0x7f76bc0703d0 2024-01-21T19:47:39.654 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.660+0000 7f76c1876700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76bc0703d0 0x7f76bc100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:39.654 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.660+0000 7f76c1876700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76bc0703d0 0x7f76bc100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:39996/0 (socket says 172.21.15.169:39996) 2024-01-21T19:47:39.655 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.660+0000 7f76c1876700 1 -- 172.21.15.169:0/3255979460 learned_addr learned my addr 172.21.15.169:0/3255979460 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:39.655 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.661+0000 7f76c1876700 1 -- 172.21.15.169:0/3255979460 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76bc101230 con 0x7f76bc0703d0 2024-01-21T19:47:39.655 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.661+0000 7f76c1876700 1 --2- 172.21.15.169:0/3255979460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76bc0703d0 0x7f76bc100d40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f76b000b5f0 tx=0x7f76b0006ee0).ready entity=mon.0 client_cookie=3709d26b6f849154 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:39.658 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.661+0000 7f76c0874700 1 -- 172.21.15.169:0/3255979460 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76b000d600 con 0x7f76bc0703d0 2024-01-21T19:47:39.658 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.661+0000 7f76c0874700 1 -- 172.21.15.169:0/3255979460 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b000d760 con 0x7f76bc0703d0 2024-01-21T19:47:39.658 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.661+0000 7f76c0874700 1 -- 172.21.15.169:0/3255979460 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76b000da30 con 0x7f76bc0703d0 2024-01-21T19:47:39.658 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.661+0000 7f76c3ada700 1 -- 172.21.15.169:0/3255979460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76bc0703d0 msgr2=0x7f76bc100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:39.658 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.662+0000 7f76c3ada700 1 --2- 172.21.15.169:0/3255979460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76bc0703d0 0x7f76bc100d40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f76b000b5f0 tx=0x7f76b0006ee0).stop 2024-01-21T19:47:39.658 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.662+0000 7f76c3ada700 1 -- 172.21.15.169:0/3255979460 shutdown_connections 2024-01-21T19:47:39.658 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.662+0000 7f76c3ada700 1 --2- 172.21.15.169:0/3255979460 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76bc0703d0 0x7f76bc100d40 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:39.658 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.662+0000 7f76c3ada700 1 -- 172.21.15.169:0/3255979460 >> 172.21.15.169:0/3255979460 conn(0x7f76bc0f63c0 msgr2=0x7f76bc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:39.659 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.662+0000 7f76c3ada700 1 -- 172.21.15.169:0/3255979460 shutdown_connections 2024-01-21T19:47:39.659 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.662+0000 7f76c3ada700 1 -- 172.21.15.169:0/3255979460 wait complete. 2024-01-21T19:47:39.659 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.662+0000 7f76c3ada700 1 Processor -- start 2024-01-21T19:47:39.659 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.662+0000 7f76c3ada700 1 -- start start 2024-01-21T19:47:39.659 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.662+0000 7f76c3ada700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76bc0703d0 0x7f76bc10c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:39.659 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.662+0000 7f76c3ada700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc10c9b0 con 0x7f76bc0703d0 2024-01-21T19:47:39.659 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.663+0000 7f76c1876700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76bc0703d0 0x7f76bc10c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:39.659 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.663+0000 7f76c1876700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76bc0703d0 0x7f76bc10c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40008/0 (socket says 172.21.15.169:40008) 2024-01-21T19:47:39.659 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.663+0000 7f76c1876700 1 -- 172.21.15.169:0/1108585217 learned_addr learned my addr 172.21.15.169:0/1108585217 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:39.659 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.663+0000 7f76c1876700 1 -- 172.21.15.169:0/1108585217 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b000b200 con 0x7f76bc0703d0 2024-01-21T19:47:39.659 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.663+0000 7f76c1876700 1 --2- 172.21.15.169:0/1108585217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76bc0703d0 0x7f76bc10c4c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f76b0006c70 tx=0x7f76b0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:39.659 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.663+0000 7f76ae7fc700 1 -- 172.21.15.169:0/1108585217 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76b0011b40 con 0x7f76bc0703d0 2024-01-21T19:47:39.659 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.663+0000 7f76ae7fc700 1 -- 172.21.15.169:0/1108585217 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b0011ca0 con 0x7f76bc0703d0 2024-01-21T19:47:39.659 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.663+0000 7f76ae7fc700 1 -- 172.21.15.169:0/1108585217 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76b001d430 con 0x7f76bc0703d0 2024-01-21T19:47:39.660 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.663+0000 7f76c3ada700 1 -- 172.21.15.169:0/1108585217 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76bc10cbb0 con 0x7f76bc0703d0 2024-01-21T19:47:39.660 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.663+0000 7f76c3ada700 1 -- 172.21.15.169:0/1108585217 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76bc10cfc0 con 0x7f76bc0703d0 2024-01-21T19:47:39.660 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.663+0000 7f76ae7fc700 1 -- 172.21.15.169:0/1108585217 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f76b000f070 con 0x7f76bc0703d0 2024-01-21T19:47:39.660 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.663+0000 7f76ae7fc700 1 -- 172.21.15.169:0/1108585217 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f76b0027460 con 0x7f76bc0703d0 2024-01-21T19:47:39.660 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.664+0000 7f76c3ada700 1 -- 172.21.15.169:0/1108585217 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76bc045ad0 con 0x7f76bc0703d0 2024-01-21T19:47:39.660 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.666+0000 7f76ae7fc700 1 -- 172.21.15.169:0/1108585217 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f76b0016070 con 0x7f76bc0703d0 2024-01-21T19:47:39.666 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:39 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38408]: cluster 2024-01-21T19:47:39.229594+0000 mon.smithi169 (mon.0) 1 : cluster [INF] mon.smithi169 is new leader, mons smithi169 in quorum (ranks 0) 2024-01-21T19:47:39.728 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.732+0000 7f76c3ada700 1 -- 172.21.15.169:0/1108585217 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f76bc105f90 con 0x7f76bc0703d0 2024-01-21T19:47:39.728 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.732+0000 7f76ae7fc700 1 -- 172.21.15.169:0/1108585217 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f76b0036070 con 0x7f76bc0703d0 2024-01-21T19:47:39.728 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: cluster: 2024-01-21T19:47:39.728 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: id: df2470f2-b895-11ee-95b0-87774f69a715 2024-01-21T19:47:39.728 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-01-21T19:47:39.728 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.728 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: services: 2024-01-21T19:47:39.729 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi169 (age 0.500329s) 2024-01-21T19:47:39.729 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: mgr: no daemons active 2024-01-21T19:47:39.729 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-01-21T19:47:39.729 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.729 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: data: 2024-01-21T19:47:39.729 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-01-21T19:47:39.729 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-01-21T19:47:39.729 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-01-21T19:47:39.729 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: pgs: 2024-01-21T19:47:39.729 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.729 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.733+0000 7f76c3ada700 1 -- 172.21.15.169:0/1108585217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76bc0703d0 msgr2=0x7f76bc10c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:39.729 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.733+0000 7f76c3ada700 1 --2- 172.21.15.169:0/1108585217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76bc0703d0 0x7f76bc10c4c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f76b0006c70 tx=0x7f76b0016000).stop 2024-01-21T19:47:39.729 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.733+0000 7f76c3ada700 1 -- 172.21.15.169:0/1108585217 shutdown_connections 2024-01-21T19:47:39.729 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.733+0000 7f76c3ada700 1 --2- 172.21.15.169:0/1108585217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f76bc0703d0 0x7f76bc10c4c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:39.729 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.733+0000 7f76c3ada700 1 -- 172.21.15.169:0/1108585217 >> 172.21.15.169:0/1108585217 conn(0x7f76bc0f63c0 msgr2=0x7f76bc0f8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:39.730 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.733+0000 7f76c3ada700 1 -- 172.21.15.169:0/1108585217 shutdown_connections 2024-01-21T19:47:39.730 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:39.733+0000 7f76c3ada700 1 -- 172.21.15.169:0/1108585217 wait complete. 2024-01-21T19:47:40.076 INFO:teuthology.orchestra.run.smithi169.stderr:mon is available 2024-01-21T19:47:40.076 INFO:teuthology.orchestra.run.smithi169.stderr:Assimilating anything we can from ceph.conf... 2024-01-21T19:47:40.076 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z -v /var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/mon.smithi169:/var/lib/ceph/mon/ceph-smithi169:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi169/config 2024-01-21T19:47:40.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:40 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38408]: cluster 2024-01-21T19:47:39.233459+0000 mon.smithi169 (mon.0) 2 : cluster [INF] mon.smithi169 is new leader, mons smithi169 in quorum (ranks 0) 2024-01-21T19:47:40.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:40 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38408]: cluster 2024-01-21T19:47:39.233847+0000 mon.smithi169 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi169=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0]} 2024-01-21T19:47:40.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:40 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38408]: cluster 2024-01-21T19:47:39.236272+0000 mon.smithi169 (mon.0) 4 : cluster [DBG] fsmap 2024-01-21T19:47:40.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:40 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38408]: cluster 2024-01-21T19:47:39.238221+0000 mon.smithi169 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-21T19:47:40.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:40 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38408]: cluster 2024-01-21T19:47:39.238609+0000 mon.smithi169 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-01-21T19:47:40.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:40 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38408]: audit 2024-01-21T19:47:39.733744+0000 mon.smithi169 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.169:0/1108585217' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-21T19:47:40.462 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.466+0000 7fc04bc14700 1 Processor -- start 2024-01-21T19:47:40.462 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.466+0000 7fc04bc14700 1 -- start start 2024-01-21T19:47:40.462 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.466+0000 7fc04bc14700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0440fbca0 0x7fc0440fc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:40.462 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.466+0000 7fc04bc14700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0440fc5b0 con 0x7fc0440fbca0 2024-01-21T19:47:40.462 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.466+0000 7fc0499b0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0440fbca0 0x7fc0440fc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:40.462 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.466+0000 7fc0499b0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0440fbca0 0x7fc0440fc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40014/0 (socket says 172.21.15.169:40014) 2024-01-21T19:47:40.462 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.466+0000 7fc0499b0700 1 -- 172.21.15.169:0/1353159224 learned_addr learned my addr 172.21.15.169:0/1353159224 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:40.462 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.467+0000 7fc0499b0700 1 -- 172.21.15.169:0/1353159224 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0440fc6f0 con 0x7fc0440fbca0 2024-01-21T19:47:40.462 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.467+0000 7fc0499b0700 1 --2- 172.21.15.169:0/1353159224 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0440fbca0 0x7fc0440fc0c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fc03400b5f0 tx=0x7fc034006ee0).ready entity=mon.0 client_cookie=1d73762618d63a7e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:40.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.467+0000 7fc0489ae700 1 -- 172.21.15.169:0/1353159224 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc03400d600 con 0x7fc0440fbca0 2024-01-21T19:47:40.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.467+0000 7fc0489ae700 1 -- 172.21.15.169:0/1353159224 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03400d760 con 0x7fc0440fbca0 2024-01-21T19:47:40.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.467+0000 7fc0489ae700 1 -- 172.21.15.169:0/1353159224 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc03400da30 con 0x7fc0440fbca0 2024-01-21T19:47:40.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.467+0000 7fc04bc14700 1 -- 172.21.15.169:0/1353159224 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0440fbca0 msgr2=0x7fc0440fc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:40.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.467+0000 7fc04bc14700 1 --2- 172.21.15.169:0/1353159224 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0440fbca0 0x7fc0440fc0c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fc03400b5f0 tx=0x7fc034006ee0).stop 2024-01-21T19:47:40.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.468+0000 7fc04bc14700 1 -- 172.21.15.169:0/1353159224 shutdown_connections 2024-01-21T19:47:40.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.468+0000 7fc04bc14700 1 --2- 172.21.15.169:0/1353159224 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0440fbca0 0x7fc0440fc0c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:40.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.468+0000 7fc04bc14700 1 -- 172.21.15.169:0/1353159224 >> 172.21.15.169:0/1353159224 conn(0x7fc0440f74c0 msgr2=0x7fc0440f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:40.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.468+0000 7fc04bc14700 1 -- 172.21.15.169:0/1353159224 shutdown_connections 2024-01-21T19:47:40.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.468+0000 7fc04bc14700 1 -- 172.21.15.169:0/1353159224 wait complete. 2024-01-21T19:47:40.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.468+0000 7fc04bc14700 1 Processor -- start 2024-01-21T19:47:40.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.468+0000 7fc04bc14700 1 -- start start 2024-01-21T19:47:40.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.468+0000 7fc04bc14700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0440fbca0 0x7fc044072570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:40.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.468+0000 7fc04bc14700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0440709f0 con 0x7fc0440fbca0 2024-01-21T19:47:40.464 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.468+0000 7fc0499b0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0440fbca0 0x7fc044072570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:40.464 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.468+0000 7fc0499b0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0440fbca0 0x7fc044072570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40020/0 (socket says 172.21.15.169:40020) 2024-01-21T19:47:40.464 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.468+0000 7fc0499b0700 1 -- 172.21.15.169:0/1976738290 learned_addr learned my addr 172.21.15.169:0/1976738290 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:40.465 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.469+0000 7fc0499b0700 1 -- 172.21.15.169:0/1976738290 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03400b200 con 0x7fc0440fbca0 2024-01-21T19:47:40.465 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.469+0000 7fc0499b0700 1 --2- 172.21.15.169:0/1976738290 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0440fbca0 0x7fc044072570 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fc034006c70 tx=0x7fc034016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:40.465 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.469+0000 7fc0327fc700 1 -- 172.21.15.169:0/1976738290 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc034011b40 con 0x7fc0440fbca0 2024-01-21T19:47:40.465 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.469+0000 7fc0327fc700 1 -- 172.21.15.169:0/1976738290 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc034011ca0 con 0x7fc0440fbca0 2024-01-21T19:47:40.465 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.469+0000 7fc0327fc700 1 -- 172.21.15.169:0/1976738290 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc03401d430 con 0x7fc0440fbca0 2024-01-21T19:47:40.465 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.469+0000 7fc04bc14700 1 -- 172.21.15.169:0/1976738290 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc044070b90 con 0x7fc0440fbca0 2024-01-21T19:47:40.465 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.469+0000 7fc04bc14700 1 -- 172.21.15.169:0/1976738290 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc044070fa0 con 0x7fc0440fbca0 2024-01-21T19:47:40.465 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.470+0000 7fc0327fc700 1 -- 172.21.15.169:0/1976738290 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fc03400f070 con 0x7fc0440fbca0 2024-01-21T19:47:40.465 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.470+0000 7fc0327fc700 1 -- 172.21.15.169:0/1976738290 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fc034027460 con 0x7fc0440fbca0 2024-01-21T19:47:40.465 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.470+0000 7fc04bc14700 1 -- 172.21.15.169:0/1976738290 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc044059100 con 0x7fc0440fbca0 2024-01-21T19:47:40.466 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.472+0000 7fc0327fc700 1 -- 172.21.15.169:0/1976738290 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fc034016070 con 0x7fc0440fbca0 2024-01-21T19:47:40.530 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.536+0000 7fc04bc14700 1 -- 172.21.15.169:0/1976738290 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fc044045ad0 con 0x7fc0440fbca0 2024-01-21T19:47:40.535 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.540+0000 7fc0327fc700 1 -- 172.21.15.169:0/1976738290 <== mon.0 v2:172.21.15.169:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc03401d620 con 0x7fc0440fbca0 2024-01-21T19:47:40.535 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.540+0000 7fc0327fc700 1 -- 172.21.15.169:0/1976738290 <== mon.0 v2:172.21.15.169:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+466 (secure 0 0 0) 0x7fc034030840 con 0x7fc0440fbca0 2024-01-21T19:47:40.535 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.536 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: [global] 2024-01-21T19:47:40.536 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: fsid = df2470f2-b895-11ee-95b0-87774f69a715 2024-01-21T19:47:40.536 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.169:3300,v1:172.21.15.169:6789] 2024-01-21T19:47:40.536 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-21T19:47:40.536 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-21T19:47:40.536 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-21T19:47:40.536 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-21T19:47:40.536 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.536 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: [mgr] 2024-01-21T19:47:40.536 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-21T19:47:40.536 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.536 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: [osd] 2024-01-21T19:47:40.536 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-01-21T19:47:40.536 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-01-21T19:47:40.536 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-01-21T19:47:40.537 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-21T19:47:40.537 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-01-21T19:47:40.537 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-21T19:47:40.537 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.540+0000 7fc04bc14700 1 -- 172.21.15.169:0/1976738290 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0440fbca0 msgr2=0x7fc044072570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:40.537 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.540+0000 7fc04bc14700 1 --2- 172.21.15.169:0/1976738290 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0440fbca0 0x7fc044072570 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fc034006c70 tx=0x7fc034016000).stop 2024-01-21T19:47:40.537 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.540+0000 7fc04bc14700 1 -- 172.21.15.169:0/1976738290 shutdown_connections 2024-01-21T19:47:40.537 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.540+0000 7fc04bc14700 1 --2- 172.21.15.169:0/1976738290 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0440fbca0 0x7fc044072570 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:40.537 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.540+0000 7fc04bc14700 1 -- 172.21.15.169:0/1976738290 >> 172.21.15.169:0/1976738290 conn(0x7fc0440f74c0 msgr2=0x7fc0440fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:40.537 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.541+0000 7fc04bc14700 1 -- 172.21.15.169:0/1976738290 shutdown_connections 2024-01-21T19:47:40.537 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:40.541+0000 7fc04bc14700 1 -- 172.21.15.169:0/1976738290 wait complete. 2024-01-21T19:47:40.875 INFO:teuthology.orchestra.run.smithi169.stderr:Generating new minimal ceph.conf... 2024-01-21T19:47:40.875 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z -v /var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/mon.smithi169:/var/lib/ceph/mon/ceph-smithi169:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi169/config 2024-01-21T19:47:41.253 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.257+0000 7f8d2f73a700 1 Processor -- start 2024-01-21T19:47:41.253 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.257+0000 7f8d2f73a700 1 -- start start 2024-01-21T19:47:41.253 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.257+0000 7f8d2f73a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d280703d0 0x7f8d28100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:41.253 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.257+0000 7f8d2f73a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d280fc730 con 0x7f8d280703d0 2024-01-21T19:47:41.253 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.258+0000 7f8d2d4d6700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d280703d0 0x7f8d28100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:41.253 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.258+0000 7f8d2d4d6700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d280703d0 0x7f8d28100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40034/0 (socket says 172.21.15.169:40034) 2024-01-21T19:47:41.253 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.258+0000 7f8d2d4d6700 1 -- 172.21.15.169:0/3238101111 learned_addr learned my addr 172.21.15.169:0/3238101111 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:41.253 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.258+0000 7f8d2d4d6700 1 -- 172.21.15.169:0/3238101111 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d28101230 con 0x7f8d280703d0 2024-01-21T19:47:41.253 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.258+0000 7f8d2d4d6700 1 --2- 172.21.15.169:0/3238101111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d280703d0 0x7f8d28100d40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f8d1800b5f0 tx=0x7f8d18006ee0).ready entity=mon.0 client_cookie=5aebda90f909555 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:41.253 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.258+0000 7f8d17fff700 1 -- 172.21.15.169:0/3238101111 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d1800d600 con 0x7f8d280703d0 2024-01-21T19:47:41.253 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.258+0000 7f8d17fff700 1 -- 172.21.15.169:0/3238101111 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8d1800dbd0 con 0x7f8d280703d0 2024-01-21T19:47:41.253 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.258+0000 7f8d17fff700 1 -- 172.21.15.169:0/3238101111 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d18011c00 con 0x7f8d280703d0 2024-01-21T19:47:41.254 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.259+0000 7f8d2f73a700 1 -- 172.21.15.169:0/3238101111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d280703d0 msgr2=0x7f8d28100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:41.254 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.259+0000 7f8d2f73a700 1 --2- 172.21.15.169:0/3238101111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d280703d0 0x7f8d28100d40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f8d1800b5f0 tx=0x7f8d18006ee0).stop 2024-01-21T19:47:41.254 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.259+0000 7f8d2f73a700 1 -- 172.21.15.169:0/3238101111 shutdown_connections 2024-01-21T19:47:41.254 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.259+0000 7f8d2f73a700 1 --2- 172.21.15.169:0/3238101111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d280703d0 0x7f8d28100d40 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:41.254 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.259+0000 7f8d2f73a700 1 -- 172.21.15.169:0/3238101111 >> 172.21.15.169:0/3238101111 conn(0x7f8d280f63c0 msgr2=0x7f8d280f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:41.254 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.259+0000 7f8d2f73a700 1 -- 172.21.15.169:0/3238101111 shutdown_connections 2024-01-21T19:47:41.254 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.259+0000 7f8d2f73a700 1 -- 172.21.15.169:0/3238101111 wait complete. 2024-01-21T19:47:41.255 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.260+0000 7f8d2f73a700 1 Processor -- start 2024-01-21T19:47:41.255 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.260+0000 7f8d2f73a700 1 -- start start 2024-01-21T19:47:41.255 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.260+0000 7f8d2f73a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d280703d0 0x7f8d2810cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:41.255 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.260+0000 7f8d2f73a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2810d100 con 0x7f8d280703d0 2024-01-21T19:47:41.255 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.260+0000 7f8d2d4d6700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d280703d0 0x7f8d2810cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:41.255 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.260+0000 7f8d2d4d6700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d280703d0 0x7f8d2810cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40040/0 (socket says 172.21.15.169:40040) 2024-01-21T19:47:41.255 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.260+0000 7f8d2d4d6700 1 -- 172.21.15.169:0/2073591974 learned_addr learned my addr 172.21.15.169:0/2073591974 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:41.255 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.260+0000 7f8d2d4d6700 1 -- 172.21.15.169:0/2073591974 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1800b200 con 0x7f8d280703d0 2024-01-21T19:47:41.255 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.261+0000 7f8d2d4d6700 1 --2- 172.21.15.169:0/2073591974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d280703d0 0x7f8d2810cc10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8d1800e040 tx=0x7f8d18006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:41.256 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.261+0000 7f8d15ffb700 1 -- 172.21.15.169:0/2073591974 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d1800d9f0 con 0x7f8d280703d0 2024-01-21T19:47:41.256 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.261+0000 7f8d15ffb700 1 -- 172.21.15.169:0/2073591974 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8d1801a9f0 con 0x7f8d280703d0 2024-01-21T19:47:41.256 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.261+0000 7f8d15ffb700 1 -- 172.21.15.169:0/2073591974 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d1801b8c0 con 0x7f8d280703d0 2024-01-21T19:47:41.256 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.261+0000 7f8d2f73a700 1 -- 172.21.15.169:0/2073591974 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2810d300 con 0x7f8d280703d0 2024-01-21T19:47:41.256 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.261+0000 7f8d2f73a700 1 -- 172.21.15.169:0/2073591974 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2810d710 con 0x7f8d280703d0 2024-01-21T19:47:41.256 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.261+0000 7f8d15ffb700 1 -- 172.21.15.169:0/2073591974 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f8d1800f070 con 0x7f8d280703d0 2024-01-21T19:47:41.256 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.261+0000 7f8d15ffb700 1 -- 172.21.15.169:0/2073591974 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f8d18025ae0 con 0x7f8d280703d0 2024-01-21T19:47:41.256 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.262+0000 7f8d2f73a700 1 -- 172.21.15.169:0/2073591974 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d28045ad0 con 0x7f8d280703d0 2024-01-21T19:47:41.257 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.264+0000 7f8d15ffb700 1 -- 172.21.15.169:0/2073591974 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f8d18016070 con 0x7f8d280703d0 2024-01-21T19:47:41.321 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.328+0000 7f8d2f73a700 1 -- 172.21.15.169:0/2073591974 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f8d28101d90 con 0x7f8d280703d0 2024-01-21T19:47:41.322 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.328+0000 7f8d15ffb700 1 -- 172.21.15.169:0/2073591974 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f8d1801bd70 con 0x7f8d280703d0 2024-01-21T19:47:41.323 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.329+0000 7f8d2f73a700 1 -- 172.21.15.169:0/2073591974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d280703d0 msgr2=0x7f8d2810cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:41.323 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.329+0000 7f8d2f73a700 1 --2- 172.21.15.169:0/2073591974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d280703d0 0x7f8d2810cc10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8d1800e040 tx=0x7f8d18006ee0).stop 2024-01-21T19:47:41.323 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.329+0000 7f8d2f73a700 1 -- 172.21.15.169:0/2073591974 shutdown_connections 2024-01-21T19:47:41.323 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.329+0000 7f8d2f73a700 1 --2- 172.21.15.169:0/2073591974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d280703d0 0x7f8d2810cc10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:41.323 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.329+0000 7f8d2f73a700 1 -- 172.21.15.169:0/2073591974 >> 172.21.15.169:0/2073591974 conn(0x7f8d280f63c0 msgr2=0x7f8d280f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:41.323 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.329+0000 7f8d2f73a700 1 -- 172.21.15.169:0/2073591974 shutdown_connections 2024-01-21T19:47:41.323 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:41.329+0000 7f8d2f73a700 1 -- 172.21.15.169:0/2073591974 wait complete. 2024-01-21T19:47:41.635 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:41 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38408]: audit 2024-01-21T19:47:40.537935+0000 mon.smithi169 (mon.0) 8 : audit [INF] from='client.? 172.21.15.169:0/1976738290' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-21T19:47:41.636 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:41 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38408]: audit 2024-01-21T19:47:40.540968+0000 mon.smithi169 (mon.0) 9 : audit [INF] from='client.? 172.21.15.169:0/1976738290' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-21T19:47:41.684 INFO:teuthology.orchestra.run.smithi169.stderr:Restarting the monitor... 2024-01-21T19:47:41.684 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl restart ceph-df2470f2-b895-11ee-95b0-87774f69a715@mon.smithi169 2024-01-21T19:47:41.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:41 smithi169 systemd[1]: Stopping Ceph mon.smithi169 for df2470f2-b895-11ee-95b0-87774f69a715... 2024-01-21T19:47:41.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:41 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38408]: debug 2024-01-21T19:47:41.780+0000 7f276e4a5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi169 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-21T19:47:41.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:41 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38408]: debug 2024-01-21T19:47:41.780+0000 7f276e4a5700 -1 mon.smithi169@0(leader) e1 *** Got Signal Terminated *** 2024-01-21T19:47:42.416 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 podman[38732]: ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169 2024-01-21T19:47:42.416 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 systemd[1]: ceph-df2470f2-b895-11ee-95b0-87774f69a715@mon.smithi169.service: Succeeded. 2024-01-21T19:47:42.416 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 systemd[1]: Stopped Ceph mon.smithi169 for df2470f2-b895-11ee-95b0-87774f69a715. 2024-01-21T19:47:42.416 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 systemd[1]: Starting Ceph mon.smithi169 for df2470f2-b895-11ee-95b0-87774f69a715... 2024-01-21T19:47:42.609 INFO:teuthology.orchestra.run.smithi169.stderr:Setting mon public_network to 172.21.0.0/20 2024-01-21T19:47:42.609 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-01-21T19:47:42.918 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.607+0000 7f8821b4e700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-21T19:47:42.918 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.607+0000 7f8821b4e700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-01-21T19:47:42.918 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.607+0000 7f8821b4e700 0 pidfile_write: ignore empty --pid-file 2024-01-21T19:47:42.918 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.611+0000 7f8821b4e700 0 load: jerasure load: lrc load: isa 2024-01-21T19:47:42.919 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: RocksDB version: 6.8.1 2024-01-21T19:47:42.919 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.919 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-21T19:47:42.919 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Compile date May 13 2021 2024-01-21T19:47:42.919 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: DB SUMMARY 2024-01-21T19:47:42.919 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.919 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: CURRENT file: CURRENT 2024-01-21T19:47:42.919 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.919 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: IDENTITY file: IDENTITY 2024-01-21T19:47:42.919 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.919 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 bash[38805]: 319b09670abdeff9da005900eb363729c1738f387aabb05463f010d83a439099 2024-01-21T19:47:42.919 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-01-21T19:47:42.919 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.919 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi169/store.db dir, Total Num: 1, files: 000004.sst 2024-01-21T19:47:42.920 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.920 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi169/store.db: 000006.log size: 92938 ; 2024-01-21T19:47:42.920 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.920 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.error_if_exists: 0 2024-01-21T19:47:42.920 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.create_if_missing: 0 2024-01-21T19:47:42.920 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.paranoid_checks: 1 2024-01-21T19:47:42.920 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.env: 0x560270e981c0 2024-01-21T19:47:42.920 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.fs: Posix File System 2024-01-21T19:47:42.920 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.info_log: 0x560272bd5960 2024-01-21T19:47:42.920 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-21T19:47:42.920 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.statistics: (nil) 2024-01-21T19:47:42.920 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.use_fsync: 0 2024-01-21T19:47:42.920 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_log_file_size: 0 2024-01-21T19:47:42.920 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-21T19:47:42.921 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-21T19:47:42.921 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-21T19:47:42.921 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-21T19:47:42.921 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.allow_fallocate: 1 2024-01-21T19:47:42.921 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-21T19:47:42.921 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-21T19:47:42.921 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.use_direct_reads: 0 2024-01-21T19:47:42.921 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-21T19:47:42.921 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-21T19:47:42.921 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.db_log_dir: 2024-01-21T19:47:42.921 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi169/store.db 2024-01-21T19:47:42.921 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-21T19:47:42.921 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_subcompactions: 1 2024-01-21T19:47:42.921 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_background_flushes: -1 2024-01-21T19:47:42.921 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-21T19:47:42.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-21T19:47:42.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-21T19:47:42.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-21T19:47:42.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-21T19:47:42.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-21T19:47:42.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-21T19:47:42.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.write_buffer_manager: 0x560272bdbfb0 2024-01-21T19:47:42.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-21T19:47:42.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-21T19:47:42.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-21T19:47:42.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-21T19:47:42.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-21T19:47:42.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-21T19:47:42.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-21T19:47:42.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-21T19:47:42.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-21T19:47:42.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.unordered_write: 0 2024-01-21T19:47:42.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-21T19:47:42.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-21T19:47:42.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-21T19:47:42.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-21T19:47:42.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.row_cache: None 2024-01-21T19:47:42.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.wal_filter: None 2024-01-21T19:47:42.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-21T19:47:42.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-21T19:47:42.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.preserve_deletes: 0 2024-01-21T19:47:42.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.two_write_queues: 0 2024-01-21T19:47:42.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-21T19:47:42.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.atomic_flush: 0 2024-01-21T19:47:42.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-21T19:47:42.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-21T19:47:42.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-21T19:47:42.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.log_readahead_size: 0 2024-01-21T19:47:42.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-21T19:47:42.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_background_jobs: 2 2024-01-21T19:47:42.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_background_compactions: -1 2024-01-21T19:47:42.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-21T19:47:42.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-21T19:47:42.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-21T19:47:42.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-21T19:47:42.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-21T19:47:42.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-21T19:47:42.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-21T19:47:42.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-21T19:47:42.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_open_files: -1 2024-01-21T19:47:42.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-21T19:47:42.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-21T19:47:42.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-21T19:47:42.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-21T19:47:42.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Compression algorithms supported: 2024-01-21T19:47:42.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-21T19:47:42.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: kZSTD supported: 0 2024-01-21T19:47:42.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: kXpressCompression supported: 0 2024-01-21T19:47:42.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-21T19:47:42.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: kLZ4Compression supported: 1 2024-01-21T19:47:42.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: kBZip2Compression supported: 0 2024-01-21T19:47:42.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: kZlibCompression supported: 1 2024-01-21T19:47:42.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: kSnappyCompression supported: 1 2024-01-21T19:47:42.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-21T19:47:42.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi169/store.db/MANIFEST-000005 2024-01-21T19:47:42.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-21T19:47:42.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-21T19:47:42.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.merge_operator: 2024-01-21T19:47:42.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.compaction_filter: None 2024-01-21T19:47:42.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-21T19:47:42.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-21T19:47:42.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-21T19:47:42.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560272aec128) 2024-01-21T19:47:42.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cache_index_and_filter_blocks: 1 2024-01-21T19:47:42.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-21T19:47:42.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-21T19:47:42.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: pin_top_level_index_and_filter: 1 2024-01-21T19:47:42.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: index_type: 0 2024-01-21T19:47:42.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: data_block_index_type: 0 2024-01-21T19:47:42.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: index_shortening: 1 2024-01-21T19:47:42.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: data_block_hash_table_util_ratio: 0.750000 2024-01-21T19:47:42.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: hash_index_allow_collision: 1 2024-01-21T19:47:42.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: checksum: 1 2024-01-21T19:47:42.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: no_block_cache: 0 2024-01-21T19:47:42.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: block_cache: 0x560272b22f10 2024-01-21T19:47:42.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: block_cache_name: BinnedLRUCache 2024-01-21T19:47:42.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: block_cache_options: 2024-01-21T19:47:42.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: capacity : 536870912 2024-01-21T19:47:42.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: num_shard_bits : 4 2024-01-21T19:47:42.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: strict_capacity_limit : 0 2024-01-21T19:47:42.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: high_pri_pool_ratio: 0.000 2024-01-21T19:47:42.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: block_cache_compressed: (nil) 2024-01-21T19:47:42.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: persistent_cache: (nil) 2024-01-21T19:47:42.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: block_size: 4096 2024-01-21T19:47:42.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: block_size_deviation: 10 2024-01-21T19:47:42.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: block_restart_interval: 16 2024-01-21T19:47:42.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: index_block_restart_interval: 1 2024-01-21T19:47:42.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: metadata_block_size: 4096 2024-01-21T19:47:42.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: partition_filters: 0 2024-01-21T19:47:42.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: use_delta_encoding: 1 2024-01-21T19:47:42.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-21T19:47:42.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: whole_key_filtering: 1 2024-01-21T19:47:42.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: verify_compression: 0 2024-01-21T19:47:42.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: read_amp_bytes_per_bit: 0 2024-01-21T19:47:42.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: format_version: 2 2024-01-21T19:47:42.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: enable_index_compression: 1 2024-01-21T19:47:42.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: block_align: 0 2024-01-21T19:47:42.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-21T19:47:42.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-21T19:47:42.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.compression: NoCompression 2024-01-21T19:47:42.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-21T19:47:42.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-21T19:47:42.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-21T19:47:42.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.num_levels: 7 2024-01-21T19:47:42.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-21T19:47:42.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-21T19:47:42.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-21T19:47:42.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-21T19:47:42.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-21T19:47:42.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-21T19:47:42.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-21T19:47:42.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-21T19:47:42.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-21T19:47:42.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-21T19:47:42.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-21T19:47:42.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-21T19:47:42.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-21T19:47:42.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-21T19:47:42.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-21T19:47:42.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-21T19:47:42.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-21T19:47:42.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-21T19:47:42.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-21T19:47:42.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-21T19:47:42.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-21T19:47:42.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-21T19:47:42.931 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-21T19:47:42.931 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-21T19:47:42.931 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-21T19:47:42.931 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-21T19:47:42.931 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-21T19:47:42.931 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-21T19:47:42.931 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-21T19:47:42.931 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-21T19:47:42.931 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-21T19:47:42.931 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-21T19:47:42.931 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-21T19:47:42.931 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-21T19:47:42.931 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-21T19:47:42.931 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-21T19:47:42.931 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-21T19:47:42.931 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-21T19:47:42.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-21T19:47:42.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-21T19:47:42.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-21T19:47:42.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-21T19:47:42.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-21T19:47:42.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-21T19:47:42.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-21T19:47:42.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-21T19:47:42.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-21T19:47:42.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.table_properties_collectors: 2024-01-21T19:47:42.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.inplace_update_support: 0 2024-01-21T19:47:42.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-21T19:47:42.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-21T19:47:42.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-21T19:47:42.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-21T19:47:42.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.bloom_locality: 0 2024-01-21T19:47:42.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.max_successive_merges: 0 2024-01-21T19:47:42.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-21T19:47:42.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-21T19:47:42.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-21T19:47:42.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-21T19:47:42.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.ttl: 2592000 2024-01-21T19:47:42.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.612+0000 7f8821b4e700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-21T19:47:42.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.613+0000 7f8821b4e700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi169/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-21T19:47:42.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.613+0000 7f8821b4e700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-01-21T19:47:42.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.613+0000 7f8821b4e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705866462614393, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-01-21T19:47:42.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.613+0000 7f8821b4e700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-01-21T19:47:42.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.613+0000 7f8821b4e700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-21T19:47:42.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.614+0000 7f8821b4e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705866462615243, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88718, "table_properties": {"data_size": 86833, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79974, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705866462, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-21T19:47:42.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.614+0000 7f8821b4e700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-01-21T19:47:42.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.614+0000 7f8821b4e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705866462615634, "job": 1, "event": "recovery_finished"} 2024-01-21T19:47:42.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 systemd[1]: Started Ceph mon.smithi169 for df2470f2-b895-11ee-95b0-87774f69a715. 2024-01-21T19:47:42.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.615+0000 7f8821b4e700 4 rocksdb: DB pointer 0x560272be7800 2024-01-21T19:47:42.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.615+0000 7f8808966700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-21T19:47:42.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.615+0000 7f8808966700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-21T19:47:42.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: ** DB Stats ** 2024-01-21T19:47:42.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T19:47:42.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-21T19:47:42.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-21T19:47:42.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T19:47:42.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-21T19:47:42.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-21T19:47:42.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T19:47:42.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: ** Compaction Stats [default] ** 2024-01-21T19:47:42.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:47:42.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:47:42.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: L0 2/0 88.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.4 0.00 0.00 1 0.001 0 0 2024-01-21T19:47:42.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Sum 2/0 88.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.4 0.00 0.00 1 0.001 0 0 2024-01-21T19:47:42.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.4 0.00 0.00 1 0.001 0 0 2024-01-21T19:47:42.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: ** Compaction Stats [default] ** 2024-01-21T19:47:42.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:47:42.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:47:42.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 146.4 0.00 0.00 1 0.001 0 0 2024-01-21T19:47:42.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T19:47:42.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T19:47:42.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T19:47:42.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T19:47:42.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T19:47:42.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: AddFile(Keys): cumulative 0, interval 0 2024-01-21T19:47:42.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Cumulative compaction: 0.00 GB write, 28.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:47:42.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Interval compaction: 0.00 GB write, 28.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:47:42.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T19:47:42.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: ** File Read Latency Histogram By Level [default] ** 2024-01-21T19:47:42.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: ** Compaction Stats [default] ** 2024-01-21T19:47:42.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:47:42.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:47:42.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: L0 2/0 88.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.4 0.00 0.00 1 0.001 0 0 2024-01-21T19:47:42.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Sum 2/0 88.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.4 0.00 0.00 1 0.001 0 0 2024-01-21T19:47:42.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T19:47:42.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: ** Compaction Stats [default] ** 2024-01-21T19:47:42.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:47:42.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:47:42.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 146.4 0.00 0.00 1 0.001 0 0 2024-01-21T19:47:42.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T19:47:42.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T19:47:42.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T19:47:42.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T19:47:42.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T19:47:42.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: AddFile(Keys): cumulative 0, interval 0 2024-01-21T19:47:42.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Cumulative compaction: 0.00 GB write, 27.86 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:47:42.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:47:42.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T19:47:42.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: ** File Read Latency Histogram By Level [default] ** 2024-01-21T19:47:42.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.616+0000 7f8821b4e700 0 starting mon.smithi169 rank 0 at public addrs [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] at bind addrs [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi169 fsid df2470f2-b895-11ee-95b0-87774f69a715 2024-01-21T19:47:42.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.617+0000 7f8821b4e700 1 mon.smithi169@-1(???) e1 preinit fsid df2470f2-b895-11ee-95b0-87774f69a715 2024-01-21T19:47:42.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.617+0000 7f8821b4e700 0 mon.smithi169@-1(???).mds e1 new map 2024-01-21T19:47:42.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.617+0000 7f8821b4e700 0 mon.smithi169@-1(???).mds e1 print_map 2024-01-21T19:47:42.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: e1 2024-01-21T19:47:42.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-21T19:47:42.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-21T19:47:42.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: legacy client fscid: -1 2024-01-21T19:47:42.939 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.939 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: No filesystems configured 2024-01-21T19:47:42.939 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: 2024-01-21T19:47:42.939 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.617+0000 7f8821b4e700 0 mon.smithi169@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-21T19:47:42.939 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.617+0000 7f8821b4e700 0 mon.smithi169@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T19:47:42.939 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.617+0000 7f8821b4e700 0 mon.smithi169@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T19:47:42.939 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.617+0000 7f8821b4e700 0 mon.smithi169@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T19:47:42.939 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:40.537935+0000 mon.smithi169 (mon.0) 8 : audit [INF] from='client.? 172.21.15.169:0/1976738290' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-21T19:47:42.939 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:40.540968+0000 mon.smithi169 (mon.0) 9 : audit [INF] from='client.? 172.21.15.169:0/1976738290' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-21T19:47:42.939 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.617+0000 7f8821b4e700 1 mon.smithi169@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-21T19:47:42.939 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.618+0000 7f8821b4e700 20 expand_channel_meta expand map: {default=false} 2024-01-21T19:47:42.939 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.618+0000 7f8821b4e700 20 expand_channel_meta from 'false' to 'false' 2024-01-21T19:47:42.939 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.618+0000 7f8821b4e700 20 expand_channel_meta expanded map: {default=false} 2024-01-21T19:47:42.939 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.618+0000 7f8821b4e700 20 expand_channel_meta expand map: {default=info} 2024-01-21T19:47:42.939 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.618+0000 7f8821b4e700 20 expand_channel_meta from 'info' to 'info' 2024-01-21T19:47:42.940 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.618+0000 7f8821b4e700 20 expand_channel_meta expanded map: {default=info} 2024-01-21T19:47:42.940 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.618+0000 7f8821b4e700 20 expand_channel_meta expand map: {default=daemon} 2024-01-21T19:47:42.940 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.618+0000 7f8821b4e700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-01-21T19:47:42.940 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.618+0000 7f8821b4e700 20 expand_channel_meta expanded map: {default=daemon} 2024-01-21T19:47:42.940 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.618+0000 7f8821b4e700 20 expand_channel_meta expand map: {default=debug} 2024-01-21T19:47:42.940 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.618+0000 7f8821b4e700 20 expand_channel_meta from 'debug' to 'debug' 2024-01-21T19:47:42.940 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:47:42.618+0000 7f8821b4e700 20 expand_channel_meta expanded map: {default=debug} 2024-01-21T19:47:42.940 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:47:42.633583+0000 mon.smithi169 (mon.0) 1 : cluster [INF] mon.smithi169 is new leader, mons smithi169 in quorum (ranks 0) 2024-01-21T19:47:42.940 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:47:42.633670+0000 mon.smithi169 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi169=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0]} 2024-01-21T19:47:42.940 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:47:42.633715+0000 mon.smithi169 (mon.0) 3 : cluster [DBG] fsmap 2024-01-21T19:47:42.940 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:47:42.633746+0000 mon.smithi169 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-21T19:47:42.940 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:47:42.634175+0000 mon.smithi169 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-01-21T19:47:42.978 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.980+0000 7fd8864e7700 1 Processor -- start 2024-01-21T19:47:42.978 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.980+0000 7fd8864e7700 1 -- start start 2024-01-21T19:47:42.979 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.980+0000 7fd8864e7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd880070430 0x7fd8800fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:42.979 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.980+0000 7fd8864e7700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8800fedd0 con 0x7fd880070430 2024-01-21T19:47:42.979 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.981+0000 7fd87ffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd880070430 0x7fd8800fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:42.979 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.981+0000 7fd87ffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd880070430 0x7fd8800fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40042/0 (socket says 172.21.15.169:40042) 2024-01-21T19:47:42.979 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.981+0000 7fd87ffff700 1 -- 172.21.15.169:0/1638507836 learned_addr learned my addr 172.21.15.169:0/1638507836 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:42.979 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.981+0000 7fd87ffff700 1 -- 172.21.15.169:0/1638507836 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8800fef10 con 0x7fd880070430 2024-01-21T19:47:42.979 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.981+0000 7fd87ffff700 1 --2- 172.21.15.169:0/1638507836 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd880070430 0x7fd8800fe8e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fd87000b5f0 tx=0x7fd870006ee0).ready entity=mon.0 client_cookie=8d0f0a24788bf849 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:42.979 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.981+0000 7fd87effd700 1 -- 172.21.15.169:0/1638507836 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd87000d600 con 0x7fd880070430 2024-01-21T19:47:42.979 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.982+0000 7fd87effd700 1 -- 172.21.15.169:0/1638507836 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd87000dbd0 con 0x7fd880070430 2024-01-21T19:47:42.979 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.982+0000 7fd87effd700 1 -- 172.21.15.169:0/1638507836 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd870011c00 con 0x7fd880070430 2024-01-21T19:47:42.979 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.982+0000 7fd8864e7700 1 -- 172.21.15.169:0/1638507836 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd880070430 msgr2=0x7fd8800fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:42.979 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.982+0000 7fd8864e7700 1 --2- 172.21.15.169:0/1638507836 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd880070430 0x7fd8800fe8e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fd87000b5f0 tx=0x7fd870006ee0).stop 2024-01-21T19:47:42.979 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.982+0000 7fd8864e7700 1 -- 172.21.15.169:0/1638507836 shutdown_connections 2024-01-21T19:47:42.979 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.982+0000 7fd8864e7700 1 --2- 172.21.15.169:0/1638507836 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd880070430 0x7fd8800fe8e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:42.979 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.982+0000 7fd8864e7700 1 -- 172.21.15.169:0/1638507836 >> 172.21.15.169:0/1638507836 conn(0x7fd8800f63c0 msgr2=0x7fd8800f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:42.980 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.982+0000 7fd8864e7700 1 -- 172.21.15.169:0/1638507836 shutdown_connections 2024-01-21T19:47:42.980 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.982+0000 7fd8864e7700 1 -- 172.21.15.169:0/1638507836 wait complete. 2024-01-21T19:47:42.980 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.983+0000 7fd8864e7700 1 Processor -- start 2024-01-21T19:47:42.980 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.983+0000 7fd8864e7700 1 -- start start 2024-01-21T19:47:42.980 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.983+0000 7fd8864e7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd880070430 0x7fd8800fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:42.980 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.983+0000 7fd8864e7700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8800fa7e0 con 0x7fd880070430 2024-01-21T19:47:42.980 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.983+0000 7fd87ffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd880070430 0x7fd8800fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:42.980 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.983+0000 7fd87ffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd880070430 0x7fd8800fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40054/0 (socket says 172.21.15.169:40054) 2024-01-21T19:47:42.980 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.983+0000 7fd87ffff700 1 -- 172.21.15.169:0/3878859560 learned_addr learned my addr 172.21.15.169:0/3878859560 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:42.980 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.983+0000 7fd87ffff700 1 -- 172.21.15.169:0/3878859560 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87000b200 con 0x7fd880070430 2024-01-21T19:47:42.980 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.984+0000 7fd87ffff700 1 --2- 172.21.15.169:0/3878859560 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd880070430 0x7fd8800fc360 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fd87000b190 tx=0x7fd870006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:42.980 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.984+0000 7fd87cff9700 1 -- 172.21.15.169:0/3878859560 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd87001b410 con 0x7fd880070430 2024-01-21T19:47:42.980 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.984+0000 7fd87cff9700 1 -- 172.21.15.169:0/3878859560 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd87001b9e0 con 0x7fd880070430 2024-01-21T19:47:42.980 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.984+0000 7fd8864e7700 1 -- 172.21.15.169:0/3878859560 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8800fa980 con 0x7fd880070430 2024-01-21T19:47:42.981 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.984+0000 7fd87cff9700 1 -- 172.21.15.169:0/3878859560 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd87001a880 con 0x7fd880070430 2024-01-21T19:47:42.981 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.984+0000 7fd8864e7700 1 -- 172.21.15.169:0/3878859560 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8800fad90 con 0x7fd880070430 2024-01-21T19:47:42.981 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.984+0000 7fd87cff9700 1 -- 172.21.15.169:0/3878859560 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fd87000f070 con 0x7fd880070430 2024-01-21T19:47:42.981 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.984+0000 7fd87cff9700 1 -- 172.21.15.169:0/3878859560 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd870025b70 con 0x7fd880070430 2024-01-21T19:47:42.981 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.985+0000 7fd8864e7700 1 -- 172.21.15.169:0/3878859560 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd880045ad0 con 0x7fd880070430 2024-01-21T19:47:42.981 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:42.987+0000 7fd87cff9700 1 -- 172.21.15.169:0/3878859560 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fd870016070 con 0x7fd880070430 2024-01-21T19:47:43.045 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:43.051+0000 7fd8864e7700 1 -- 172.21.15.169:0/3878859560 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fd8800ffb90 con 0x7fd880070430 2024-01-21T19:47:43.049 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:43.056+0000 7fd87cff9700 1 -- 172.21.15.169:0/3878859560 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd870025db0 con 0x7fd880070430 2024-01-21T19:47:43.051 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:43.057+0000 7fd8864e7700 1 -- 172.21.15.169:0/3878859560 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd880070430 msgr2=0x7fd8800fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:43.051 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:43.057+0000 7fd8864e7700 1 --2- 172.21.15.169:0/3878859560 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd880070430 0x7fd8800fc360 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fd87000b190 tx=0x7fd870006ee0).stop 2024-01-21T19:47:43.051 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:43.057+0000 7fd8864e7700 1 -- 172.21.15.169:0/3878859560 shutdown_connections 2024-01-21T19:47:43.051 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:43.057+0000 7fd8864e7700 1 --2- 172.21.15.169:0/3878859560 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd880070430 0x7fd8800fc360 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:43.051 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:43.057+0000 7fd8864e7700 1 -- 172.21.15.169:0/3878859560 >> 172.21.15.169:0/3878859560 conn(0x7fd8800f63c0 msgr2=0x7fd8800f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:43.052 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:43.057+0000 7fd8864e7700 1 -- 172.21.15.169:0/3878859560 shutdown_connections 2024-01-21T19:47:43.052 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:43.057+0000 7fd8864e7700 1 -- 172.21.15.169:0/3878859560 wait complete. 2024-01-21T19:47:43.416 INFO:teuthology.orchestra.run.smithi169.stderr:Wrote config to /etc/ceph/ceph.conf 2024-01-21T19:47:43.417 INFO:teuthology.orchestra.run.smithi169.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-21T19:47:43.417 INFO:teuthology.orchestra.run.smithi169.stderr:Creating mgr... 2024-01-21T19:47:43.417 INFO:teuthology.orchestra.run.smithi169.stderr:Verifying port 9283 ... 2024-01-21T19:47:43.419 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl daemon-reload 2024-01-21T19:47:43.514 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl stop ceph-df2470f2-b895-11ee-95b0-87774f69a715@mgr.smithi169.espgue 2024-01-21T19:47:43.543 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl reset-failed ceph-df2470f2-b895-11ee-95b0-87774f69a715@mgr.smithi169.espgue 2024-01-21T19:47:43.549 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: Failed to reset failed state of unit ceph-df2470f2-b895-11ee-95b0-87774f69a715@mgr.smithi169.espgue.service: Unit ceph-df2470f2-b895-11ee-95b0-87774f69a715@mgr.smithi169.espgue.service not loaded. 2024-01-21T19:47:43.549 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl enable ceph-df2470f2-b895-11ee-95b0-87774f69a715@mgr.smithi169.espgue 2024-01-21T19:47:43.555 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: Created symlink /etc/systemd/system/ceph-df2470f2-b895-11ee-95b0-87774f69a715.target.wants/ceph-df2470f2-b895-11ee-95b0-87774f69a715@mgr.smithi169.espgue.service → /etc/systemd/system/ceph-df2470f2-b895-11ee-95b0-87774f69a715@.service. 2024-01-21T19:47:43.640 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl start ceph-df2470f2-b895-11ee-95b0-87774f69a715@mgr.smithi169.espgue 2024-01-21T19:47:43.976 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T19:47:43.985 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: disabled 2024-01-21T19:47:43.986 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T19:47:43.994 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: inactive 2024-01-21T19:47:43.995 INFO:teuthology.orchestra.run.smithi169.stderr:firewalld.service is not enabled 2024-01-21T19:47:43.995 INFO:teuthology.orchestra.run.smithi169.stderr:Not possible to enable service . firewalld.service is not available 2024-01-21T19:47:43.995 INFO:teuthology.orchestra.run.smithi169.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-21T19:47:43.995 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T19:47:44.003 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: disabled 2024-01-21T19:47:44.004 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T19:47:44.012 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: inactive 2024-01-21T19:47:44.013 INFO:teuthology.orchestra.run.smithi169.stderr:firewalld.service is not enabled 2024-01-21T19:47:44.013 INFO:teuthology.orchestra.run.smithi169.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-01-21T19:47:44.013 INFO:teuthology.orchestra.run.smithi169.stderr:Waiting for mgr to start... 2024-01-21T19:47:44.013 INFO:teuthology.orchestra.run.smithi169.stderr:Waiting for mgr... 2024-01-21T19:47:44.013 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T19:47:44.374 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.379+0000 7f9897847700 1 Processor -- start 2024-01-21T19:47:44.375 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.379+0000 7f9897847700 1 -- start start 2024-01-21T19:47:44.375 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.379+0000 7f9897847700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9890070430 0x7f98900fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:44.375 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.379+0000 7f9897847700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98900fedd0 con 0x7f9890070430 2024-01-21T19:47:44.375 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.380+0000 7f98955e3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9890070430 0x7f98900fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:44.375 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.380+0000 7f98955e3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9890070430 0x7f98900fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40070/0 (socket says 172.21.15.169:40070) 2024-01-21T19:47:44.375 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.380+0000 7f98955e3700 1 -- 172.21.15.169:0/3415679579 learned_addr learned my addr 172.21.15.169:0/3415679579 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:44.375 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.380+0000 7f98955e3700 1 -- 172.21.15.169:0/3415679579 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98900fef10 con 0x7f9890070430 2024-01-21T19:47:44.375 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.380+0000 7f98955e3700 1 --2- 172.21.15.169:0/3415679579 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9890070430 0x7f98900fe8e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f988c00b5f0 tx=0x7f988c006ee0).ready entity=mon.0 client_cookie=b4b13678ea1c7083 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:44.375 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.380+0000 7f987ffff700 1 -- 172.21.15.169:0/3415679579 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f988c00d600 con 0x7f9890070430 2024-01-21T19:47:44.375 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.380+0000 7f987ffff700 1 -- 172.21.15.169:0/3415679579 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f988c00dbd0 con 0x7f9890070430 2024-01-21T19:47:44.375 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.380+0000 7f987ffff700 1 -- 172.21.15.169:0/3415679579 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f988c011c00 con 0x7f9890070430 2024-01-21T19:47:44.375 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.381+0000 7f9897847700 1 -- 172.21.15.169:0/3415679579 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9890070430 msgr2=0x7f98900fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:44.376 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.381+0000 7f9897847700 1 --2- 172.21.15.169:0/3415679579 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9890070430 0x7f98900fe8e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f988c00b5f0 tx=0x7f988c006ee0).stop 2024-01-21T19:47:44.376 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.381+0000 7f9897847700 1 -- 172.21.15.169:0/3415679579 shutdown_connections 2024-01-21T19:47:44.376 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.381+0000 7f9897847700 1 --2- 172.21.15.169:0/3415679579 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9890070430 0x7f98900fe8e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:44.376 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.381+0000 7f9897847700 1 -- 172.21.15.169:0/3415679579 >> 172.21.15.169:0/3415679579 conn(0x7f98900f63c0 msgr2=0x7f98900f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:44.377 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.382+0000 7f9897847700 1 -- 172.21.15.169:0/3415679579 shutdown_connections 2024-01-21T19:47:44.377 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.382+0000 7f9897847700 1 -- 172.21.15.169:0/3415679579 wait complete. 2024-01-21T19:47:44.377 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.382+0000 7f9897847700 1 Processor -- start 2024-01-21T19:47:44.377 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.382+0000 7f9897847700 1 -- start start 2024-01-21T19:47:44.377 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.382+0000 7f9897847700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9890070430 0x7f989010aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:44.378 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.382+0000 7f9897847700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989010aef0 con 0x7f9890070430 2024-01-21T19:47:44.378 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.383+0000 7f98955e3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9890070430 0x7f989010aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:44.378 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.383+0000 7f98955e3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9890070430 0x7f989010aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40082/0 (socket says 172.21.15.169:40082) 2024-01-21T19:47:44.378 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.383+0000 7f98955e3700 1 -- 172.21.15.169:0/2779015225 learned_addr learned my addr 172.21.15.169:0/2779015225 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:44.378 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.383+0000 7f98955e3700 1 -- 172.21.15.169:0/2779015225 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c00b200 con 0x7f9890070430 2024-01-21T19:47:44.378 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.383+0000 7f98955e3700 1 --2- 172.21.15.169:0/2779015225 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9890070430 0x7f989010aa00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f988c00b190 tx=0x7f988c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:44.378 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.383+0000 7f987dffb700 1 -- 172.21.15.169:0/2779015225 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f988c01b410 con 0x7f9890070430 2024-01-21T19:47:44.378 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.383+0000 7f987dffb700 1 -- 172.21.15.169:0/2779015225 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f988c01b9e0 con 0x7f9890070430 2024-01-21T19:47:44.378 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.383+0000 7f9897847700 1 -- 172.21.15.169:0/2779015225 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f989010b0f0 con 0x7f9890070430 2024-01-21T19:47:44.378 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.383+0000 7f987dffb700 1 -- 172.21.15.169:0/2779015225 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f988c01a880 con 0x7f9890070430 2024-01-21T19:47:44.378 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.383+0000 7f9897847700 1 -- 172.21.15.169:0/2779015225 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f989010b500 con 0x7f9890070430 2024-01-21T19:47:44.379 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.385+0000 7f987dffb700 1 -- 172.21.15.169:0/2779015225 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f988c00f070 con 0x7f9890070430 2024-01-21T19:47:44.379 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.385+0000 7f987dffb700 1 -- 172.21.15.169:0/2779015225 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f988c025820 con 0x7f9890070430 2024-01-21T19:47:44.379 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.385+0000 7f9897847700 1 -- 172.21.15.169:0/2779015225 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9890045ad0 con 0x7f9890070430 2024-01-21T19:47:44.380 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.387+0000 7f987dffb700 1 -- 172.21.15.169:0/2779015225 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f988c016070 con 0x7f9890070430 2024-01-21T19:47:44.416 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:44 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:43.057667+0000 mon.smithi169 (mon.0) 6 : audit [INF] from='client.? 172.21.15.169:0/3878859560' entity='client.admin' 2024-01-21T19:47:44.450 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.455+0000 7f9897847700 1 -- 172.21.15.169:0/2779015225 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f98901048f0 con 0x7f9890070430 2024-01-21T19:47:44.450 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.456+0000 7f987dffb700 1 -- 172.21.15.169:0/2779015225 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f988c01aaf0 con 0x7f9890070430 2024-01-21T19:47:44.450 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.450 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: { 2024-01-21T19:47:44.450 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "fsid": "df2470f2-b895-11ee-95b0-87774f69a715", 2024-01-21T19:47:44.451 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "health": { 2024-01-21T19:47:44.451 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T19:47:44.451 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T19:47:44.451 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T19:47:44.451 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: }, 2024-01-21T19:47:44.451 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T19:47:44.459 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T19:47:44.459 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 0 2024-01-21T19:47:44.459 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: ], 2024-01-21T19:47:44.459 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T19:47:44.460 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "smithi169" 2024-01-21T19:47:44.460 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: ], 2024-01-21T19:47:44.460 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "quorum_age": 1, 2024-01-21T19:47:44.460 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T19:47:44.460 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:47:44.460 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T19:47:44.461 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T19:47:44.461 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: }, 2024-01-21T19:47:44.461 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T19:47:44.461 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:47:44.461 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T19:47:44.461 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T19:47:44.461 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T19:47:44.461 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T19:47:44.462 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T19:47:44.462 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T19:47:44.462 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: }, 2024-01-21T19:47:44.462 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T19:47:44.462 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T19:47:44.462 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T19:47:44.462 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T19:47:44.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T19:47:44.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T19:47:44.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T19:47:44.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T19:47:44.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T19:47:44.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: }, 2024-01-21T19:47:44.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T19:47:44.463 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:47:44.464 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T19:47:44.464 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T19:47:44.464 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: }, 2024-01-21T19:47:44.464 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T19:47:44.464 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "available": false, 2024-01-21T19:47:44.465 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T19:47:44.465 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T19:47:44.465 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "iostat", 2024-01-21T19:47:44.465 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "restful" 2024-01-21T19:47:44.465 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: ], 2024-01-21T19:47:44.465 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "services": {} 2024-01-21T19:47:44.466 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: }, 2024-01-21T19:47:44.466 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T19:47:44.466 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:47:44.466 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "modified": "2024-01-21T19:47:39.234573+0000", 2024-01-21T19:47:44.466 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "services": {} 2024-01-21T19:47:44.467 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: }, 2024-01-21T19:47:44.467 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T19:47:44.467 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: } 2024-01-21T19:47:44.467 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.457+0000 7f9897847700 1 -- 172.21.15.169:0/2779015225 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9890070430 msgr2=0x7f989010aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:44.467 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.457+0000 7f9897847700 1 --2- 172.21.15.169:0/2779015225 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9890070430 0x7f989010aa00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f988c00b190 tx=0x7f988c006ee0).stop 2024-01-21T19:47:44.467 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.457+0000 7f9897847700 1 -- 172.21.15.169:0/2779015225 shutdown_connections 2024-01-21T19:47:44.468 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.457+0000 7f9897847700 1 --2- 172.21.15.169:0/2779015225 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9890070430 0x7f989010aa00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:44.468 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.457+0000 7f9897847700 1 -- 172.21.15.169:0/2779015225 >> 172.21.15.169:0/2779015225 conn(0x7f98900f63c0 msgr2=0x7f98900f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:44.468 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.457+0000 7f9897847700 1 -- 172.21.15.169:0/2779015225 shutdown_connections 2024-01-21T19:47:44.468 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:44.457+0000 7f9897847700 1 -- 172.21.15.169:0/2779015225 wait complete. 2024-01-21T19:47:44.825 INFO:teuthology.orchestra.run.smithi169.stderr:mgr not available, waiting (1/15)... 2024-01-21T19:47:45.416 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:45 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:44.457050+0000 mon.smithi169 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.169:0/2779015225' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T19:47:46.827 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T19:47:47.249 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.255+0000 7f66a7bdc700 1 Processor -- start 2024-01-21T19:47:47.249 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.255+0000 7f66a7bdc700 1 -- start start 2024-01-21T19:47:47.250 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.255+0000 7f66a7bdc700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66a00fab80 0x7f66a00fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:47.250 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.255+0000 7f66a7bdc700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a00fb490 con 0x7f66a00fab80 2024-01-21T19:47:47.250 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.255+0000 7f66a5978700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66a00fab80 0x7f66a00fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:47.250 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.256+0000 7f66a5978700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66a00fab80 0x7f66a00fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40086/0 (socket says 172.21.15.169:40086) 2024-01-21T19:47:47.250 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.256+0000 7f66a5978700 1 -- 172.21.15.169:0/672755932 learned_addr learned my addr 172.21.15.169:0/672755932 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:47.251 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.256+0000 7f66a5978700 1 -- 172.21.15.169:0/672755932 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a00fb5d0 con 0x7f66a00fab80 2024-01-21T19:47:47.251 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.256+0000 7f66a5978700 1 --2- 172.21.15.169:0/672755932 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66a00fab80 0x7f66a00fafa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f669c00b870 tx=0x7f669c006f70).ready entity=mon.0 client_cookie=45eb599a24e302d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:47.252 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.256+0000 7f66a4976700 1 -- 172.21.15.169:0/672755932 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f669c00da20 con 0x7f66a00fab80 2024-01-21T19:47:47.252 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.256+0000 7f66a4976700 1 -- 172.21.15.169:0/672755932 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f669c010470 con 0x7f66a00fab80 2024-01-21T19:47:47.252 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.256+0000 7f66a4976700 1 -- 172.21.15.169:0/672755932 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f669c00f5b0 con 0x7f66a00fab80 2024-01-21T19:47:47.252 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.257+0000 7f66a7bdc700 1 -- 172.21.15.169:0/672755932 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66a00fab80 msgr2=0x7f66a00fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:47.252 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.257+0000 7f66a7bdc700 1 --2- 172.21.15.169:0/672755932 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66a00fab80 0x7f66a00fafa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f669c00b870 tx=0x7f669c006f70).stop 2024-01-21T19:47:47.252 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.257+0000 7f66a7bdc700 1 -- 172.21.15.169:0/672755932 shutdown_connections 2024-01-21T19:47:47.252 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.257+0000 7f66a7bdc700 1 --2- 172.21.15.169:0/672755932 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66a00fab80 0x7f66a00fafa0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:47.252 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.257+0000 7f66a7bdc700 1 -- 172.21.15.169:0/672755932 >> 172.21.15.169:0/672755932 conn(0x7f66a00f63c0 msgr2=0x7f66a00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:47.252 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.257+0000 7f66a7bdc700 1 -- 172.21.15.169:0/672755932 shutdown_connections 2024-01-21T19:47:47.252 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.257+0000 7f66a7bdc700 1 -- 172.21.15.169:0/672755932 wait complete. 2024-01-21T19:47:47.253 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.257+0000 7f66a7bdc700 1 Processor -- start 2024-01-21T19:47:47.254 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.257+0000 7f66a7bdc700 1 -- start start 2024-01-21T19:47:47.254 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.258+0000 7f66a7bdc700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66a00fab80 0x7f66a0108770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:47.254 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.258+0000 7f66a7bdc700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a0108c60 con 0x7f66a00fab80 2024-01-21T19:47:47.254 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.258+0000 7f66a5978700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66a00fab80 0x7f66a0108770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:47.254 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.258+0000 7f66a5978700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66a00fab80 0x7f66a0108770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:40100/0 (socket says 172.21.15.169:40100) 2024-01-21T19:47:47.254 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.258+0000 7f66a5978700 1 -- 172.21.15.169:0/480709787 learned_addr learned my addr 172.21.15.169:0/480709787 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:47.254 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.258+0000 7f66a5978700 1 -- 172.21.15.169:0/480709787 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669c00b480 con 0x7f66a00fab80 2024-01-21T19:47:47.254 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.258+0000 7f66a5978700 1 --2- 172.21.15.169:0/480709787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66a00fab80 0x7f66a0108770 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f669c006a40 tx=0x7f669c006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:47.254 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.258+0000 7f668e7fc700 1 -- 172.21.15.169:0/480709787 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f669c00f3d0 con 0x7f66a00fab80 2024-01-21T19:47:47.254 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.258+0000 7f66a7bdc700 1 -- 172.21.15.169:0/480709787 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a0108e60 con 0x7f66a00fab80 2024-01-21T19:47:47.254 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.258+0000 7f668e7fc700 1 -- 172.21.15.169:0/480709787 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f669c00fef0 con 0x7f66a00fab80 2024-01-21T19:47:47.254 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.258+0000 7f668e7fc700 1 -- 172.21.15.169:0/480709787 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f669c019d40 con 0x7f66a00fab80 2024-01-21T19:47:47.255 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.258+0000 7f66a7bdc700 1 -- 172.21.15.169:0/480709787 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a0109270 con 0x7f66a00fab80 2024-01-21T19:47:47.255 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.258+0000 7f668e7fc700 1 -- 172.21.15.169:0/480709787 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f669c02b060 con 0x7f66a00fab80 2024-01-21T19:47:47.255 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.259+0000 7f668e7fc700 1 -- 172.21.15.169:0/480709787 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f669c02ea60 con 0x7f66a00fab80 2024-01-21T19:47:47.255 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.259+0000 7f66a7bdc700 1 -- 172.21.15.169:0/480709787 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a0045ad0 con 0x7f66a00fab80 2024-01-21T19:47:47.255 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.261+0000 7f668e7fc700 1 -- 172.21.15.169:0/480709787 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f669c029070 con 0x7f66a00fab80 2024-01-21T19:47:47.330 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.330+0000 7f66a7bdc700 1 -- 172.21.15.169:0/480709787 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f66a01025d0 con 0x7f66a00fab80 2024-01-21T19:47:47.330 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.330+0000 7f668e7fc700 1 -- 172.21.15.169:0/480709787 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f669c014070 con 0x7f66a00fab80 2024-01-21T19:47:47.330 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: { 2024-01-21T19:47:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "fsid": "df2470f2-b895-11ee-95b0-87774f69a715", 2024-01-21T19:47:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "health": { 2024-01-21T19:47:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T19:47:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T19:47:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T19:47:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: }, 2024-01-21T19:47:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T19:47:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T19:47:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 0 2024-01-21T19:47:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: ], 2024-01-21T19:47:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T19:47:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "smithi169" 2024-01-21T19:47:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: ], 2024-01-21T19:47:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "quorum_age": 4, 2024-01-21T19:47:47.331 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T19:47:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:47:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T19:47:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T19:47:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: }, 2024-01-21T19:47:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T19:47:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:47:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T19:47:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T19:47:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T19:47:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T19:47:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T19:47:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T19:47:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: }, 2024-01-21T19:47:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T19:47:47.332 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T19:47:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T19:47:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T19:47:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T19:47:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T19:47:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T19:47:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T19:47:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T19:47:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: }, 2024-01-21T19:47:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T19:47:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:47:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T19:47:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T19:47:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: }, 2024-01-21T19:47:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T19:47:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "available": false, 2024-01-21T19:47:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T19:47:47.333 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T19:47:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "iostat", 2024-01-21T19:47:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "restful" 2024-01-21T19:47:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: ], 2024-01-21T19:47:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "services": {} 2024-01-21T19:47:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: }, 2024-01-21T19:47:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T19:47:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:47:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "modified": "2024-01-21T19:47:39.234573+0000", 2024-01-21T19:47:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "services": {} 2024-01-21T19:47:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: }, 2024-01-21T19:47:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T19:47:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: } 2024-01-21T19:47:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.332+0000 7f66a7bdc700 1 -- 172.21.15.169:0/480709787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66a00fab80 msgr2=0x7f66a0108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.332+0000 7f66a7bdc700 1 --2- 172.21.15.169:0/480709787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66a00fab80 0x7f66a0108770 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f669c006a40 tx=0x7f669c006f70).stop 2024-01-21T19:47:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.332+0000 7f66a7bdc700 1 -- 172.21.15.169:0/480709787 shutdown_connections 2024-01-21T19:47:47.334 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.332+0000 7f66a7bdc700 1 --2- 172.21.15.169:0/480709787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f66a00fab80 0x7f66a0108770 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:47.335 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.332+0000 7f66a7bdc700 1 -- 172.21.15.169:0/480709787 >> 172.21.15.169:0/480709787 conn(0x7f66a00f63c0 msgr2=0x7f66a00ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:47.335 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.332+0000 7f66a7bdc700 1 -- 172.21.15.169:0/480709787 shutdown_connections 2024-01-21T19:47:47.335 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:47.332+0000 7f66a7bdc700 1 -- 172.21.15.169:0/480709787 wait complete. 2024-01-21T19:47:47.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:47 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:47.331527+0000 mon.smithi169 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.169:0/480709787' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T19:47:47.708 INFO:teuthology.orchestra.run.smithi169.stderr:mgr not available, waiting (2/15)... 2024-01-21T19:47:49.416 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:47:48.942029+0000 mon.smithi169 (mon.0) 9 : cluster [INF] Activating manager daemon smithi169.espgue 2024-01-21T19:47:49.416 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:47:48.944736+0000 mon.smithi169 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi169.espgue(active, starting, since 0.00282111s) 2024-01-21T19:47:49.416 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:48.945409+0000 mon.smithi169 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.169:0/1279341068' entity='mgr.smithi169.espgue' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T19:47:49.416 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:48.945615+0000 mon.smithi169 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.169:0/1279341068' entity='mgr.smithi169.espgue' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T19:47:49.417 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:48.945752+0000 mon.smithi169 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.169:0/1279341068' entity='mgr.smithi169.espgue' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T19:47:49.417 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:48.945864+0000 mon.smithi169 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.169:0/1279341068' entity='mgr.smithi169.espgue' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-01-21T19:47:49.417 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:48.946174+0000 mon.smithi169 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.169:0/1279341068' entity='mgr.smithi169.espgue' cmd=[{"prefix": "mgr metadata", "who": "smithi169.espgue", "id": "smithi169.espgue"}]: dispatch 2024-01-21T19:47:49.417 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:47:48.959364+0000 mon.smithi169 (mon.0) 16 : cluster [INF] Manager daemon smithi169.espgue is now available 2024-01-21T19:47:49.417 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:48.968725+0000 mon.smithi169 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.169:0/1279341068' entity='mgr.smithi169.espgue' 2024-01-21T19:47:49.417 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:48.971865+0000 mon.smithi169 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.169:0/1279341068' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/mirror_snapshot_schedule"}]: dispatch 2024-01-21T19:47:49.417 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:48.972967+0000 mon.smithi169 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.169:0/1279341068' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/trash_purge_schedule"}]: dispatch 2024-01-21T19:47:49.417 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:48.977414+0000 mon.smithi169 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.169:0/1279341068' entity='mgr.smithi169.espgue' 2024-01-21T19:47:49.417 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:48.979223+0000 mon.smithi169 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.169:0/1279341068' entity='mgr.smithi169.espgue' 2024-01-21T19:47:49.710 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T19:47:50.066 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.068+0000 7fd1450a4700 1 Processor -- start 2024-01-21T19:47:50.066 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.068+0000 7fd1450a4700 1 -- start start 2024-01-21T19:47:50.066 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.068+0000 7fd1450a4700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd140070350 0x7fd1400fe9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:50.067 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.068+0000 7fd1450a4700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd140070770 con 0x7fd140070350 2024-01-21T19:47:50.067 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.068+0000 7fd13ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd140070350 0x7fd1400fe9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:50.067 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.069+0000 7fd13ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd140070350 0x7fd1400fe9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36930/0 (socket says 172.21.15.169:36930) 2024-01-21T19:47:50.067 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.069+0000 7fd13ed9d700 1 -- 172.21.15.169:0/4282403578 learned_addr learned my addr 172.21.15.169:0/4282403578 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:50.067 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.069+0000 7fd13ed9d700 1 -- 172.21.15.169:0/4282403578 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1400feeb0 con 0x7fd140070350 2024-01-21T19:47:50.067 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.069+0000 7fd13ed9d700 1 --2- 172.21.15.169:0/4282403578 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd140070350 0x7fd1400fe9c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fd12800b5f0 tx=0x7fd128006ee0).ready entity=mon.0 client_cookie=8bb707928bca82b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:50.067 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.069+0000 7fd13dd9b700 1 -- 172.21.15.169:0/4282403578 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd12800d600 con 0x7fd140070350 2024-01-21T19:47:50.067 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.069+0000 7fd13dd9b700 1 -- 172.21.15.169:0/4282403578 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd12800dbd0 con 0x7fd140070350 2024-01-21T19:47:50.067 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.069+0000 7fd13dd9b700 1 -- 172.21.15.169:0/4282403578 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd128011c00 con 0x7fd140070350 2024-01-21T19:47:50.067 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.070+0000 7fd1450a4700 1 -- 172.21.15.169:0/4282403578 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd140070350 msgr2=0x7fd1400fe9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:50.067 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.070+0000 7fd1450a4700 1 --2- 172.21.15.169:0/4282403578 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd140070350 0x7fd1400fe9c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fd12800b5f0 tx=0x7fd128006ee0).stop 2024-01-21T19:47:50.067 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.070+0000 7fd1450a4700 1 -- 172.21.15.169:0/4282403578 shutdown_connections 2024-01-21T19:47:50.067 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.070+0000 7fd1450a4700 1 --2- 172.21.15.169:0/4282403578 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd140070350 0x7fd1400fe9c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:50.067 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.070+0000 7fd1450a4700 1 -- 172.21.15.169:0/4282403578 >> 172.21.15.169:0/4282403578 conn(0x7fd1400f63c0 msgr2=0x7fd1400f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:50.068 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.070+0000 7fd1450a4700 1 -- 172.21.15.169:0/4282403578 shutdown_connections 2024-01-21T19:47:50.068 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.070+0000 7fd1450a4700 1 -- 172.21.15.169:0/4282403578 wait complete. 2024-01-21T19:47:50.068 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.070+0000 7fd1450a4700 1 Processor -- start 2024-01-21T19:47:50.068 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.070+0000 7fd1450a4700 1 -- start start 2024-01-21T19:47:50.068 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.071+0000 7fd1450a4700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd140070350 0x7fd14010aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:50.068 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.071+0000 7fd1450a4700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14010aef0 con 0x7fd140070350 2024-01-21T19:47:50.068 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.071+0000 7fd13ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd140070350 0x7fd14010aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:50.068 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.071+0000 7fd13ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd140070350 0x7fd14010aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36938/0 (socket says 172.21.15.169:36938) 2024-01-21T19:47:50.068 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.071+0000 7fd13ed9d700 1 -- 172.21.15.169:0/502871850 learned_addr learned my addr 172.21.15.169:0/502871850 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:50.068 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.071+0000 7fd13ed9d700 1 -- 172.21.15.169:0/502871850 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12800b200 con 0x7fd140070350 2024-01-21T19:47:50.068 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.071+0000 7fd13ed9d700 1 --2- 172.21.15.169:0/502871850 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd140070350 0x7fd14010aa00 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fd12800b1d0 tx=0x7fd128006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:50.068 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.071+0000 7fd1377fe700 1 -- 172.21.15.169:0/502871850 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd128011a20 con 0x7fd140070350 2024-01-21T19:47:50.068 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.071+0000 7fd1377fe700 1 -- 172.21.15.169:0/502871850 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd12801b880 con 0x7fd140070350 2024-01-21T19:47:50.069 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.071+0000 7fd1377fe700 1 -- 172.21.15.169:0/502871850 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd12801a770 con 0x7fd140070350 2024-01-21T19:47:50.069 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.071+0000 7fd1450a4700 1 -- 172.21.15.169:0/502871850 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd14010b0f0 con 0x7fd140070350 2024-01-21T19:47:50.069 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.071+0000 7fd1450a4700 1 -- 172.21.15.169:0/502871850 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd14010b500 con 0x7fd140070350 2024-01-21T19:47:50.069 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.072+0000 7fd1377fe700 1 -- 172.21.15.169:0/502871850 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7fd12800f070 con 0x7fd140070350 2024-01-21T19:47:50.069 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.072+0000 7fd1450a4700 1 -- 172.21.15.169:0/502871850 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd140045ad0 con 0x7fd140070350 2024-01-21T19:47:50.069 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.072+0000 7fd1377fe700 1 --2- 172.21.15.169:0/502871850 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7fd12002fcc0 0x7fd120032180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:50.069 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.073+0000 7fd1377fe700 1 -- 172.21.15.169:0/502871850 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd128040ef0 con 0x7fd140070350 2024-01-21T19:47:50.069 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.073+0000 7fd13e59c700 1 --2- 172.21.15.169:0/502871850 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7fd12002fcc0 0x7fd120032180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:50.069 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.073+0000 7fd13e59c700 1 --2- 172.21.15.169:0/502871850 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7fd12002fcc0 0x7fd120032180 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd1300096a0 tx=0x7fd130006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:50.070 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.077+0000 7fd1377fe700 1 -- 172.21.15.169:0/502871850 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd128022310 con 0x7fd140070350 2024-01-21T19:47:50.275 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.280+0000 7fd1450a4700 1 -- 172.21.15.169:0/502871850 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd1400ffb20 con 0x7fd140070350 2024-01-21T19:47:50.275 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.280+0000 7fd1377fe700 1 -- 172.21.15.169:0/502871850 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7fd1280164e0 con 0x7fd140070350 2024-01-21T19:47:50.275 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.275 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: { 2024-01-21T19:47:50.275 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "fsid": "df2470f2-b895-11ee-95b0-87774f69a715", 2024-01-21T19:47:50.275 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "health": { 2024-01-21T19:47:50.275 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T19:47:50.275 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T19:47:50.275 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T19:47:50.275 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: }, 2024-01-21T19:47:50.281 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T19:47:50.281 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T19:47:50.281 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 0 2024-01-21T19:47:50.281 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: ], 2024-01-21T19:47:50.281 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T19:47:50.281 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "smithi169" 2024-01-21T19:47:50.282 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: ], 2024-01-21T19:47:50.282 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "quorum_age": 7, 2024-01-21T19:47:50.282 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T19:47:50.282 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:47:50.282 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T19:47:50.282 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T19:47:50.282 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: }, 2024-01-21T19:47:50.282 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T19:47:50.282 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:47:50.282 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T19:47:50.282 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T19:47:50.282 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T19:47:50.282 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T19:47:50.282 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T19:47:50.282 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T19:47:50.282 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: }, 2024-01-21T19:47:50.283 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T19:47:50.283 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T19:47:50.283 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T19:47:50.283 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T19:47:50.283 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T19:47:50.283 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T19:47:50.283 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T19:47:50.283 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T19:47:50.283 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T19:47:50.283 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: }, 2024-01-21T19:47:50.283 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T19:47:50.283 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:47:50.283 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T19:47:50.283 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T19:47:50.283 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: }, 2024-01-21T19:47:50.283 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T19:47:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "available": true, 2024-01-21T19:47:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T19:47:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T19:47:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "iostat", 2024-01-21T19:47:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "restful" 2024-01-21T19:47:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: ], 2024-01-21T19:47:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "services": {} 2024-01-21T19:47:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: }, 2024-01-21T19:47:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T19:47:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:47:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "modified": "2024-01-21T19:47:39.234573+0000", 2024-01-21T19:47:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "services": {} 2024-01-21T19:47:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: }, 2024-01-21T19:47:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T19:47:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: } 2024-01-21T19:47:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.282+0000 7fd1450a4700 1 -- 172.21.15.169:0/502871850 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7fd12002fcc0 msgr2=0x7fd120032180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:50.285 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.282+0000 7fd1450a4700 1 --2- 172.21.15.169:0/502871850 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7fd12002fcc0 0x7fd120032180 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd1300096a0 tx=0x7fd130006b40).stop 2024-01-21T19:47:50.285 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.282+0000 7fd1450a4700 1 -- 172.21.15.169:0/502871850 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd140070350 msgr2=0x7fd14010aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:50.285 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.282+0000 7fd1450a4700 1 --2- 172.21.15.169:0/502871850 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd140070350 0x7fd14010aa00 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fd12800b1d0 tx=0x7fd128006ee0).stop 2024-01-21T19:47:50.285 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.282+0000 7fd1450a4700 1 -- 172.21.15.169:0/502871850 shutdown_connections 2024-01-21T19:47:50.285 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.282+0000 7fd1450a4700 1 --2- 172.21.15.169:0/502871850 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7fd12002fcc0 0x7fd120032180 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:50.285 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.282+0000 7fd1450a4700 1 --2- 172.21.15.169:0/502871850 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd140070350 0x7fd14010aa00 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:50.285 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.282+0000 7fd1450a4700 1 -- 172.21.15.169:0/502871850 >> 172.21.15.169:0/502871850 conn(0x7fd1400f63c0 msgr2=0x7fd1400f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:50.285 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.282+0000 7fd1450a4700 1 -- 172.21.15.169:0/502871850 shutdown_connections 2024-01-21T19:47:50.285 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.282+0000 7fd1450a4700 1 -- 172.21.15.169:0/502871850 wait complete. 2024-01-21T19:47:50.624 INFO:teuthology.orchestra.run.smithi169.stderr:mgr is available 2024-01-21T19:47:50.624 INFO:teuthology.orchestra.run.smithi169.stderr:Enabling cephadm module... 2024-01-21T19:47:50.624 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-01-21T19:47:50.987 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.992+0000 7f222b795700 1 Processor -- start 2024-01-21T19:47:50.988 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.992+0000 7f222b795700 1 -- start start 2024-01-21T19:47:50.988 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.992+0000 7f222b795700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22240fc2b0 0x7f22240fc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:50.988 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.992+0000 7f222b795700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22240fcbc0 con 0x7f22240fc2b0 2024-01-21T19:47:50.988 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.993+0000 7f2229531700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22240fc2b0 0x7f22240fc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:50.988 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.993+0000 7f2229531700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22240fc2b0 0x7f22240fc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36948/0 (socket says 172.21.15.169:36948) 2024-01-21T19:47:50.988 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.993+0000 7f2229531700 1 -- 172.21.15.169:0/1084705006 learned_addr learned my addr 172.21.15.169:0/1084705006 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:50.988 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.993+0000 7f2229531700 1 -- 172.21.15.169:0/1084705006 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22240fcd00 con 0x7f22240fc2b0 2024-01-21T19:47:50.988 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.993+0000 7f2229531700 1 --2- 172.21.15.169:0/1084705006 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22240fc2b0 0x7f22240fc6d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f221c009960 tx=0x7f221c0092d0).ready entity=mon.0 client_cookie=2602217313bc2e87 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:50.988 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.993+0000 7f2213fff700 1 -- 172.21.15.169:0/1084705006 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f221c00b9c0 con 0x7f22240fc2b0 2024-01-21T19:47:50.988 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.993+0000 7f2213fff700 1 -- 172.21.15.169:0/1084705006 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f221c011400 con 0x7f22240fc2b0 2024-01-21T19:47:50.988 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.993+0000 7f2213fff700 1 -- 172.21.15.169:0/1084705006 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f221c0103a0 con 0x7f22240fc2b0 2024-01-21T19:47:50.988 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.994+0000 7f222b795700 1 -- 172.21.15.169:0/1084705006 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22240fc2b0 msgr2=0x7f22240fc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:50.988 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.994+0000 7f222b795700 1 --2- 172.21.15.169:0/1084705006 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22240fc2b0 0x7f22240fc6d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f221c009960 tx=0x7f221c0092d0).stop 2024-01-21T19:47:50.988 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.994+0000 7f222b795700 1 -- 172.21.15.169:0/1084705006 shutdown_connections 2024-01-21T19:47:50.989 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.994+0000 7f222b795700 1 --2- 172.21.15.169:0/1084705006 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22240fc2b0 0x7f22240fc6d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:50.989 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.994+0000 7f222b795700 1 -- 172.21.15.169:0/1084705006 >> 172.21.15.169:0/1084705006 conn(0x7f22240f5b40 msgr2=0x7f22240f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:50.990 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.994+0000 7f222b795700 1 -- 172.21.15.169:0/1084705006 shutdown_connections 2024-01-21T19:47:50.990 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.994+0000 7f222b795700 1 -- 172.21.15.169:0/1084705006 wait complete. 2024-01-21T19:47:50.990 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.994+0000 7f222b795700 1 Processor -- start 2024-01-21T19:47:50.990 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.994+0000 7f222b795700 1 -- start start 2024-01-21T19:47:50.990 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.994+0000 7f222b795700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22240fc2b0 0x7f222410cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:50.990 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.994+0000 7f222b795700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222410d070 con 0x7f22240fc2b0 2024-01-21T19:47:50.990 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.995+0000 7f2229531700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22240fc2b0 0x7f222410cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:50.990 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.995+0000 7f2229531700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22240fc2b0 0x7f222410cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36950/0 (socket says 172.21.15.169:36950) 2024-01-21T19:47:50.990 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.995+0000 7f2229531700 1 -- 172.21.15.169:0/4125322973 learned_addr learned my addr 172.21.15.169:0/4125322973 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:50.990 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.995+0000 7f2229531700 1 -- 172.21.15.169:0/4125322973 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f221c009570 con 0x7f22240fc2b0 2024-01-21T19:47:50.990 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.995+0000 7f2229531700 1 --2- 172.21.15.169:0/4125322973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22240fc2b0 0x7f222410cb80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f221c00ba90 tx=0x7f221c0092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:50.990 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.995+0000 7f2211ffb700 1 -- 172.21.15.169:0/4125322973 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f221c010870 con 0x7f22240fc2b0 2024-01-21T19:47:50.991 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.995+0000 7f222b795700 1 -- 172.21.15.169:0/4125322973 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f222410d270 con 0x7f22240fc2b0 2024-01-21T19:47:50.991 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.995+0000 7f2211ffb700 1 -- 172.21.15.169:0/4125322973 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f221c010e40 con 0x7f22240fc2b0 2024-01-21T19:47:50.991 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.995+0000 7f2211ffb700 1 -- 172.21.15.169:0/4125322973 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f221c01ae10 con 0x7f22240fc2b0 2024-01-21T19:47:50.991 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.995+0000 7f222b795700 1 -- 172.21.15.169:0/4125322973 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f222410d680 con 0x7f22240fc2b0 2024-01-21T19:47:50.991 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.996+0000 7f2211ffb700 1 -- 172.21.15.169:0/4125322973 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f221c02a070 con 0x7f22240fc2b0 2024-01-21T19:47:50.991 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.996+0000 7f2211ffb700 1 --2- 172.21.15.169:0/4125322973 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f220c02fcb0 0x7f220c032170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:50.991 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.996+0000 7f2211ffb700 1 -- 172.21.15.169:0/4125322973 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f221c041a30 con 0x7f22240fc2b0 2024-01-21T19:47:50.991 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.996+0000 7f2228d30700 1 --2- 172.21.15.169:0/4125322973 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f220c02fcb0 0x7f220c032170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:50.991 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.996+0000 7f222b795700 1 -- 172.21.15.169:0/4125322973 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2224059100 con 0x7f22240fc2b0 2024-01-21T19:47:50.991 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:50.997+0000 7f2228d30700 1 --2- 172.21.15.169:0/4125322973 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f220c02fcb0 0x7f220c032170 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f22140096a0 tx=0x7f2214006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:50.994 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:51.001+0000 7f2211ffb700 1 -- 172.21.15.169:0/4125322973 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f221c01e080 con 0x7f22240fc2b0 2024-01-21T19:47:51.181 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:51.187+0000 7f222b795700 1 -- 172.21.15.169:0/4125322973 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f2224045ad0 con 0x7f22240fc2b0 2024-01-21T19:47:51.416 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:50 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:47:49.946950+0000 mon.smithi169 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi169.espgue(active, since 1.00503s) 2024-01-21T19:47:51.416 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:50 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:50.281776+0000 mon.smithi169 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.169:0/502871850' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T19:47:51.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:51.953+0000 7f2211ffb700 1 -- 172.21.15.169:0/4125322973 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f221c00f070 con 0x7f22240fc2b0 2024-01-21T19:47:51.947 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:51.954+0000 7f2211ffb700 1 -- 172.21.15.169:0/4125322973 <== mon.0 v2:172.21.15.169:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f221c0064a0 con 0x7f22240fc2b0 2024-01-21T19:47:51.951 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:51.956+0000 7f222b795700 1 -- 172.21.15.169:0/4125322973 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f220c02fcb0 msgr2=0x7f220c032170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:51.951 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:51.956+0000 7f222b795700 1 --2- 172.21.15.169:0/4125322973 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f220c02fcb0 0x7f220c032170 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f22140096a0 tx=0x7f2214006b40).stop 2024-01-21T19:47:51.951 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:51.956+0000 7f222b795700 1 -- 172.21.15.169:0/4125322973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22240fc2b0 msgr2=0x7f222410cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:51.951 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:51.956+0000 7f222b795700 1 --2- 172.21.15.169:0/4125322973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22240fc2b0 0x7f222410cb80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f221c00ba90 tx=0x7f221c0092d0).stop 2024-01-21T19:47:51.951 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:51.956+0000 7f222b795700 1 -- 172.21.15.169:0/4125322973 shutdown_connections 2024-01-21T19:47:51.951 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:51.956+0000 7f222b795700 1 --2- 172.21.15.169:0/4125322973 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f220c02fcb0 0x7f220c032170 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:51.951 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:51.956+0000 7f222b795700 1 --2- 172.21.15.169:0/4125322973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f22240fc2b0 0x7f222410cb80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:51.951 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:51.956+0000 7f222b795700 1 -- 172.21.15.169:0/4125322973 >> 172.21.15.169:0/4125322973 conn(0x7f22240f5b40 msgr2=0x7f22240f75b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:51.951 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:51.957+0000 7f222b795700 1 -- 172.21.15.169:0/4125322973 shutdown_connections 2024-01-21T19:47:51.951 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:51.957+0000 7f222b795700 1 -- 172.21.15.169:0/4125322973 wait complete. 2024-01-21T19:47:52.251 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:51 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:47:50.952322+0000 mon.smithi169 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi169.espgue(active, since 2s) 2024-01-21T19:47:52.251 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:51 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:51.188891+0000 mon.smithi169 (mon.0) 25 : audit [INF] from='client.? 172.21.15.169:0/4125322973' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-21T19:47:52.316 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-21T19:47:52.680 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.675+0000 7f8dddae5700 1 Processor -- start 2024-01-21T19:47:52.680 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.675+0000 7f8dddae5700 1 -- start start 2024-01-21T19:47:52.680 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.675+0000 7f8dddae5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8dd80fc110 0x7f8dd80fc530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:52.681 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.675+0000 7f8dddae5700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd80fca20 con 0x7f8dd80fc110 2024-01-21T19:47:52.681 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.676+0000 7f8ddcae3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8dd80fc110 0x7f8dd80fc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:52.681 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.676+0000 7f8ddcae3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8dd80fc110 0x7f8dd80fc530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36964/0 (socket says 172.21.15.169:36964) 2024-01-21T19:47:52.681 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.676+0000 7f8ddcae3700 1 -- 172.21.15.169:0/3784501441 learned_addr learned my addr 172.21.15.169:0/3784501441 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:52.681 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.676+0000 7f8ddcae3700 1 -- 172.21.15.169:0/3784501441 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd80fcb60 con 0x7f8dd80fc110 2024-01-21T19:47:52.681 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.676+0000 7f8ddcae3700 1 --2- 172.21.15.169:0/3784501441 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8dd80fc110 0x7f8dd80fc530 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f8dcc00b5f0 tx=0x7f8dcc006ee0).ready entity=mon.0 client_cookie=ebf907dba5055c22 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:52.681 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.676+0000 7f8dd77fe700 1 -- 172.21.15.169:0/3784501441 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dcc00d600 con 0x7f8dd80fc110 2024-01-21T19:47:52.681 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.676+0000 7f8dd77fe700 1 -- 172.21.15.169:0/3784501441 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8dcc00dbd0 con 0x7f8dd80fc110 2024-01-21T19:47:52.682 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.676+0000 7f8dd77fe700 1 -- 172.21.15.169:0/3784501441 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dcc011c00 con 0x7f8dd80fc110 2024-01-21T19:47:52.682 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.677+0000 7f8dddae5700 1 -- 172.21.15.169:0/3784501441 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8dd80fc110 msgr2=0x7f8dd80fc530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:52.682 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.677+0000 7f8dddae5700 1 --2- 172.21.15.169:0/3784501441 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8dd80fc110 0x7f8dd80fc530 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f8dcc00b5f0 tx=0x7f8dcc006ee0).stop 2024-01-21T19:47:52.682 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.677+0000 7f8dddae5700 1 -- 172.21.15.169:0/3784501441 shutdown_connections 2024-01-21T19:47:52.682 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.677+0000 7f8dddae5700 1 --2- 172.21.15.169:0/3784501441 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8dd80fc110 0x7f8dd80fc530 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:52.682 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.677+0000 7f8dddae5700 1 -- 172.21.15.169:0/3784501441 >> 172.21.15.169:0/3784501441 conn(0x7f8dd80f59a0 msgr2=0x7f8dd80f7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:52.683 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.677+0000 7f8dddae5700 1 -- 172.21.15.169:0/3784501441 shutdown_connections 2024-01-21T19:47:52.683 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.677+0000 7f8dddae5700 1 -- 172.21.15.169:0/3784501441 wait complete. 2024-01-21T19:47:52.683 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.677+0000 7f8dddae5700 1 Processor -- start 2024-01-21T19:47:52.683 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.677+0000 7f8dddae5700 1 -- start start 2024-01-21T19:47:52.683 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.678+0000 7f8dddae5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8dd80fc110 0x7f8dd810ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:52.683 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.678+0000 7f8dddae5700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd810cf20 con 0x7f8dd80fc110 2024-01-21T19:47:52.684 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.678+0000 7f8ddcae3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8dd80fc110 0x7f8dd810ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:52.684 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.678+0000 7f8ddcae3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8dd80fc110 0x7f8dd810ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36978/0 (socket says 172.21.15.169:36978) 2024-01-21T19:47:52.684 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.678+0000 7f8ddcae3700 1 -- 172.21.15.169:0/2533317166 learned_addr learned my addr 172.21.15.169:0/2533317166 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:52.684 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.678+0000 7f8ddcae3700 1 -- 172.21.15.169:0/2533317166 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dcc00b200 con 0x7f8dd80fc110 2024-01-21T19:47:52.684 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.678+0000 7f8ddcae3700 1 --2- 172.21.15.169:0/2533317166 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8dd80fc110 0x7f8dd810ca30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f8dcc00e040 tx=0x7f8dcc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:52.684 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.679+0000 7f8dd57fa700 1 -- 172.21.15.169:0/2533317166 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dcc01b410 con 0x7f8dd80fc110 2024-01-21T19:47:52.685 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.679+0000 7f8dddae5700 1 -- 172.21.15.169:0/2533317166 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd810d120 con 0x7f8dd80fc110 2024-01-21T19:47:52.685 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.679+0000 7f8dd57fa700 1 -- 172.21.15.169:0/2533317166 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8dcc01b9e0 con 0x7f8dd80fc110 2024-01-21T19:47:52.685 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.679+0000 7f8dd57fa700 1 -- 172.21.15.169:0/2533317166 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dcc01a770 con 0x7f8dd80fc110 2024-01-21T19:47:52.685 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.679+0000 7f8dddae5700 1 -- 172.21.15.169:0/2533317166 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd810d530 con 0x7f8dd80fc110 2024-01-21T19:47:52.685 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.680+0000 7f8dddae5700 1 -- 172.21.15.169:0/2533317166 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd8045ad0 con 0x7f8dd80fc110 2024-01-21T19:47:52.685 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.680+0000 7f8dd57fa700 1 -- 172.21.15.169:0/2533317166 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f8dcc00f070 con 0x7f8dd80fc110 2024-01-21T19:47:52.686 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.680+0000 7f8dd57fa700 1 --2- 172.21.15.169:0/2533317166 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f8dbc02fd50 0x7f8dbc032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:52.686 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.680+0000 7f8dd57fa700 1 -- 172.21.15.169:0/2533317166 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f8dcc040fe0 con 0x7f8dd80fc110 2024-01-21T19:47:52.686 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.680+0000 7f8dd7fff700 1 -- 172.21.15.169:0/2533317166 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f8dbc02fd50 msgr2=0x7f8dbc032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/2876308778 2024-01-21T19:47:52.686 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.680+0000 7f8dd7fff700 1 --2- 172.21.15.169:0/2533317166 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f8dbc02fd50 0x7f8dbc032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T19:47:52.686 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.684+0000 7f8dd57fa700 1 -- 172.21.15.169:0/2533317166 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8dcc024410 con 0x7f8dd80fc110 2024-01-21T19:47:52.869 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.874+0000 7f8dddae5700 1 -- 172.21.15.169:0/2533317166 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f8dd810e640 con 0x7f8dd80fc110 2024-01-21T19:47:52.869 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.875+0000 7f8dd57fa700 1 -- 172.21.15.169:0/2533317166 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f8dcc022e40 con 0x7f8dd80fc110 2024-01-21T19:47:52.869 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: { 2024-01-21T19:47:52.869 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "epoch": 5, 2024-01-21T19:47:52.869 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "available": true, 2024-01-21T19:47:52.869 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "active_name": "smithi169.espgue", 2024-01-21T19:47:52.869 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-21T19:47:52.869 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: } 2024-01-21T19:47:52.871 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.876+0000 7f8dddae5700 1 -- 172.21.15.169:0/2533317166 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f8dbc02fd50 msgr2=0x7f8dbc032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T19:47:52.871 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.876+0000 7f8dddae5700 1 --2- 172.21.15.169:0/2533317166 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f8dbc02fd50 0x7f8dbc032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:47:52.871 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.876+0000 7f8dddae5700 1 -- 172.21.15.169:0/2533317166 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8dd80fc110 msgr2=0x7f8dd810ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:52.871 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.876+0000 7f8dddae5700 1 --2- 172.21.15.169:0/2533317166 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8dd80fc110 0x7f8dd810ca30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f8dcc00e040 tx=0x7f8dcc006ee0).stop 2024-01-21T19:47:52.871 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.877+0000 7f8dddae5700 1 -- 172.21.15.169:0/2533317166 shutdown_connections 2024-01-21T19:47:52.871 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.877+0000 7f8dddae5700 1 --2- 172.21.15.169:0/2533317166 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f8dbc02fd50 0x7f8dbc032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:47:52.871 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.877+0000 7f8dddae5700 1 --2- 172.21.15.169:0/2533317166 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8dd80fc110 0x7f8dd810ca30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:52.871 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.877+0000 7f8dddae5700 1 -- 172.21.15.169:0/2533317166 >> 172.21.15.169:0/2533317166 conn(0x7f8dd80f59a0 msgr2=0x7f8dd80f74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:52.871 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.877+0000 7f8dddae5700 1 -- 172.21.15.169:0/2533317166 shutdown_connections 2024-01-21T19:47:52.871 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:52.877+0000 7f8dddae5700 1 -- 172.21.15.169:0/2533317166 wait complete. 2024-01-21T19:47:53.150 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:52 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:51.955173+0000 mon.smithi169 (mon.0) 26 : audit [INF] from='client.? 172.21.15.169:0/4125322973' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-21T19:47:53.150 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:52 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:47:51.955240+0000 mon.smithi169 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi169.espgue(active, since 3s) 2024-01-21T19:47:53.151 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:52 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:52.875997+0000 mon.smithi169 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.169:0/2533317166' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-21T19:47:53.218 INFO:teuthology.orchestra.run.smithi169.stderr:Waiting for the mgr to restart... 2024-01-21T19:47:53.218 INFO:teuthology.orchestra.run.smithi169.stderr:Waiting for mgr epoch 5... 2024-01-21T19:47:53.218 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-21T19:47:53.608 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.613+0000 7f7461f46700 1 Processor -- start 2024-01-21T19:47:53.608 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.613+0000 7f7461f46700 1 -- start start 2024-01-21T19:47:53.609 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.614+0000 7f7461f46700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f745c070430 0x7f745c0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:53.609 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.614+0000 7f7461f46700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c0fedd0 con 0x7f745c070430 2024-01-21T19:47:53.609 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.614+0000 7f745b7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f745c070430 0x7f745c0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:53.609 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.614+0000 7f745b7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f745c070430 0x7f745c0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36994/0 (socket says 172.21.15.169:36994) 2024-01-21T19:47:53.609 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.614+0000 7f745b7fe700 1 -- 172.21.15.169:0/2952718206 learned_addr learned my addr 172.21.15.169:0/2952718206 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:53.609 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.614+0000 7f745b7fe700 1 -- 172.21.15.169:0/2952718206 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745c0fef10 con 0x7f745c070430 2024-01-21T19:47:53.609 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.614+0000 7f745b7fe700 1 --2- 172.21.15.169:0/2952718206 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f745c070430 0x7f745c0fe8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f744c00b5f0 tx=0x7f744c006ee0).ready entity=mon.0 client_cookie=919268d47ff96bca server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:53.609 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.614+0000 7f745a7fc700 1 -- 172.21.15.169:0/2952718206 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f744c00d600 con 0x7f745c070430 2024-01-21T19:47:53.609 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.614+0000 7f745a7fc700 1 -- 172.21.15.169:0/2952718206 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f744c00dbd0 con 0x7f745c070430 2024-01-21T19:47:53.609 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.614+0000 7f745a7fc700 1 -- 172.21.15.169:0/2952718206 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f744c011c00 con 0x7f745c070430 2024-01-21T19:47:53.612 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.615+0000 7f7461f46700 1 -- 172.21.15.169:0/2952718206 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f745c070430 msgr2=0x7f745c0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:53.612 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.615+0000 7f7461f46700 1 --2- 172.21.15.169:0/2952718206 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f745c070430 0x7f745c0fe8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f744c00b5f0 tx=0x7f744c006ee0).stop 2024-01-21T19:47:53.612 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.615+0000 7f7461f46700 1 -- 172.21.15.169:0/2952718206 shutdown_connections 2024-01-21T19:47:53.612 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.615+0000 7f7461f46700 1 --2- 172.21.15.169:0/2952718206 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f745c070430 0x7f745c0fe8e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:53.612 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.615+0000 7f7461f46700 1 -- 172.21.15.169:0/2952718206 >> 172.21.15.169:0/2952718206 conn(0x7f745c0f63c0 msgr2=0x7f745c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:53.613 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.615+0000 7f7461f46700 1 -- 172.21.15.169:0/2952718206 shutdown_connections 2024-01-21T19:47:53.613 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.615+0000 7f7461f46700 1 -- 172.21.15.169:0/2952718206 wait complete. 2024-01-21T19:47:53.613 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.616+0000 7f7461f46700 1 Processor -- start 2024-01-21T19:47:53.613 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.616+0000 7f7461f46700 1 -- start start 2024-01-21T19:47:53.613 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.616+0000 7f7461f46700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f745c070430 0x7f745c0fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:53.613 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.616+0000 7f7461f46700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c0fa7e0 con 0x7f745c070430 2024-01-21T19:47:53.613 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.616+0000 7f745b7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f745c070430 0x7f745c0fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:53.613 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.616+0000 7f745b7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f745c070430 0x7f745c0fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:37004/0 (socket says 172.21.15.169:37004) 2024-01-21T19:47:53.613 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.616+0000 7f745b7fe700 1 -- 172.21.15.169:0/3169101378 learned_addr learned my addr 172.21.15.169:0/3169101378 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:53.613 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.617+0000 7f745b7fe700 1 -- 172.21.15.169:0/3169101378 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f744c00b200 con 0x7f745c070430 2024-01-21T19:47:53.613 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.617+0000 7f745b7fe700 1 --2- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f745c070430 0x7f745c0fc360 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f744c00e040 tx=0x7f744c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:53.613 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.617+0000 7f7443fff700 1 -- 172.21.15.169:0/3169101378 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f744c00d9f0 con 0x7f745c070430 2024-01-21T19:47:53.613 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.617+0000 7f7443fff700 1 -- 172.21.15.169:0/3169101378 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f744c01a9f0 con 0x7f745c070430 2024-01-21T19:47:53.613 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.617+0000 7f7461f46700 1 -- 172.21.15.169:0/3169101378 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f745c0fa980 con 0x7f745c070430 2024-01-21T19:47:53.614 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.617+0000 7f7443fff700 1 -- 172.21.15.169:0/3169101378 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f744c01b8c0 con 0x7f745c070430 2024-01-21T19:47:53.614 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.617+0000 7f7461f46700 1 -- 172.21.15.169:0/3169101378 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f745c0fad90 con 0x7f745c070430 2024-01-21T19:47:53.614 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.618+0000 7f7443fff700 1 -- 172.21.15.169:0/3169101378 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f744c00f070 con 0x7f745c070430 2024-01-21T19:47:53.614 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.618+0000 7f7443fff700 1 --2- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f744802fd50 0x7f7448032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:53.614 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.618+0000 7f7461f46700 1 -- 172.21.15.169:0/3169101378 --> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f745c059100 con 0x7f744802fd50 2024-01-21T19:47:53.614 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.618+0000 7f7443fff700 1 -- 172.21.15.169:0/3169101378 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f744c041290 con 0x7f745c070430 2024-01-21T19:47:53.614 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.618+0000 7f745affd700 1 -- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f744802fd50 msgr2=0x7f7448032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/2876308778 2024-01-21T19:47:53.614 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.618+0000 7f745affd700 1 --2- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f744802fd50 0x7f7448032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T19:47:53.812 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.819+0000 7f745affd700 1 -- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f744802fd50 msgr2=0x7f7448032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/2876308778 2024-01-21T19:47:53.813 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:53.819+0000 7f745affd700 1 --2- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f744802fd50 0x7f7448032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T19:47:54.213 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:54.220+0000 7f745affd700 1 -- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f744802fd50 msgr2=0x7f7448032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/2876308778 2024-01-21T19:47:54.213 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:54.220+0000 7f745affd700 1 --2- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f744802fd50 0x7f7448032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-21T19:47:55.014 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:55.021+0000 7f745affd700 1 -- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f744802fd50 msgr2=0x7f7448032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/2876308778 2024-01-21T19:47:55.014 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:55.021+0000 7f745affd700 1 --2- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f744802fd50 0x7f7448032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-21T19:47:56.615 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:56.622+0000 7f745affd700 1 -- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f744802fd50 msgr2=0x7f7448032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/2876308778 2024-01-21T19:47:56.616 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:56.622+0000 7f745affd700 1 --2- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f744802fd50 0x7f7448032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-21T19:47:57.000 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:57.006+0000 7f7443fff700 1 -- 172.21.15.169:0/3169101378 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f744c0257e0 con 0x7f745c070430 2024-01-21T19:47:57.000 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:57.006+0000 7f7443fff700 1 -- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f744802fd50 msgr2=0x7f7448032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T19:47:57.000 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:57.006+0000 7f7443fff700 1 --2- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f744802fd50 0x7f7448032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:47:57.416 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:47:56.952928+0000 mon.smithi169 (mon.0) 29 : cluster [INF] Active manager daemon smithi169.espgue restarted 2024-01-21T19:47:57.416 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:47:56.953311+0000 mon.smithi169 (mon.0) 30 : cluster [INF] Activating manager daemon smithi169.espgue 2024-01-21T19:47:57.416 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:47:56.957473+0000 mon.smithi169 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-01-21T19:47:58.010 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.016+0000 7f7443fff700 1 -- 172.21.15.169:0/3169101378 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f744c042b10 con 0x7f745c070430 2024-01-21T19:47:58.010 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.016+0000 7f7443fff700 1 --2- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f74480336c0 0x7f7448035ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:58.010 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.016+0000 7f7443fff700 1 -- 172.21.15.169:0/3169101378 --> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f745c059100 con 0x7f74480336c0 2024-01-21T19:47:58.010 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.016+0000 7f745affd700 1 --2- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f74480336c0 0x7f7448035ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:58.010 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.017+0000 7f745affd700 1 --2- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f74480336c0 0x7f7448035ab0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f7450006fd0 tx=0x7f7450006cf0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:58.011 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.017+0000 7f7443fff700 1 -- 172.21.15.169:0/3169101378 <== mgr.14114 v2:172.21.15.169:6800/2079278379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f745c059100 con 0x7f74480336c0 2024-01-21T19:47:58.019 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.023+0000 7f7461f46700 1 -- 172.21.15.169:0/3169101378 --> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f745c045ad0 con 0x7f74480336c0 2024-01-21T19:47:58.019 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.024+0000 7f7443fff700 1 -- 172.21.15.169:0/3169101378 <== mgr.14114 v2:172.21.15.169:6800/2079278379 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f745c045ad0 con 0x7f74480336c0 2024-01-21T19:47:58.019 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: { 2024-01-21T19:47:58.019 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-01-21T19:47:58.019 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "initialized": true 2024-01-21T19:47:58.019 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: } 2024-01-21T19:47:58.019 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.024+0000 7f7461f46700 1 -- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f74480336c0 msgr2=0x7f7448035ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:58.019 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.024+0000 7f7461f46700 1 --2- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f74480336c0 0x7f7448035ab0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f7450006fd0 tx=0x7f7450006cf0).stop 2024-01-21T19:47:58.019 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.024+0000 7f7461f46700 1 -- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f745c070430 msgr2=0x7f745c0fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:58.020 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.024+0000 7f7461f46700 1 --2- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f745c070430 0x7f745c0fc360 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f744c00e040 tx=0x7f744c006ee0).stop 2024-01-21T19:47:58.020 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.024+0000 7f7461f46700 1 -- 172.21.15.169:0/3169101378 shutdown_connections 2024-01-21T19:47:58.020 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.024+0000 7f7461f46700 1 --2- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f74480336c0 0x7f7448035ab0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:58.020 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.024+0000 7f7461f46700 1 --2- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f745c070430 0x7f745c0fc360 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:58.020 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.024+0000 7f7461f46700 1 --2- 172.21.15.169:0/3169101378 >> [v2:172.21.15.169:6800/2876308778,v1:172.21.15.169:6801/2876308778] conn(0x7f744802fd50 0x7f7448032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:47:58.020 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.024+0000 7f7461f46700 1 -- 172.21.15.169:0/3169101378 >> 172.21.15.169:0/3169101378 conn(0x7f745c0f63c0 msgr2=0x7f745c0f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:58.020 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.024+0000 7f7461f46700 1 -- 172.21.15.169:0/3169101378 shutdown_connections 2024-01-21T19:47:58.020 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.024+0000 7f7461f46700 1 -- 172.21.15.169:0/3169101378 wait complete. 2024-01-21T19:47:58.317 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:47:57.008677+0000 mon.smithi169 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi169.espgue(active, starting, since 0.0554393s) 2024-01-21T19:47:58.318 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:57.010056+0000 mon.smithi169 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-01-21T19:47:58.318 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:57.010609+0000 mon.smithi169 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' cmd=[{"prefix": "mgr metadata", "who": "smithi169.espgue", "id": "smithi169.espgue"}]: dispatch 2024-01-21T19:47:58.318 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:57.011442+0000 mon.smithi169 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T19:47:58.318 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:57.011590+0000 mon.smithi169 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T19:47:58.318 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:57.011699+0000 mon.smithi169 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T19:47:58.318 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:47:57.025610+0000 mon.smithi169 (mon.0) 38 : cluster [INF] Manager daemon smithi169.espgue is now available 2024-01-21T19:47:58.318 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:57.038588+0000 mon.smithi169 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' 2024-01-21T19:47:58.318 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:57.041217+0000 mon.smithi169 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' 2024-01-21T19:47:58.318 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:57.043728+0000 mon.smithi169 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' 2024-01-21T19:47:58.318 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:57.136415+0000 mon.smithi169 (mon.0) 42 : audit [INF] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' 2024-01-21T19:47:58.318 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:57.137067+0000 mon.smithi169 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:47:58.318 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:57.138620+0000 mon.smithi169 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:47:58.319 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:57.140264+0000 mon.smithi169 (mon.0) 45 : audit [DBG] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:47:58.319 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:57.150098+0000 mon.smithi169 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/mirror_snapshot_schedule"}]: dispatch 2024-01-21T19:47:58.319 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:57.151225+0000 mon.smithi169 (mon.0) 47 : audit [INF] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/trash_purge_schedule"}]: dispatch 2024-01-21T19:47:58.372 INFO:teuthology.orchestra.run.smithi169.stderr:mgr epoch 5 is available 2024-01-21T19:47:58.373 INFO:teuthology.orchestra.run.smithi169.stderr:Setting orchestrator backend to cephadm... 2024-01-21T19:47:58.373 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-01-21T19:47:58.762 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.764+0000 7f9d0778e700 1 Processor -- start 2024-01-21T19:47:58.762 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.765+0000 7f9d0778e700 1 -- start start 2024-01-21T19:47:58.762 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.765+0000 7f9d0778e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d000703f0 0x7f9d00100c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:58.763 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.765+0000 7f9d0778e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d000fc630 con 0x7f9d000703f0 2024-01-21T19:47:58.763 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.765+0000 7f9d0552a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d000703f0 0x7f9d00100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:58.763 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.765+0000 7f9d0552a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d000703f0 0x7f9d00100c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:37062/0 (socket says 172.21.15.169:37062) 2024-01-21T19:47:58.763 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.765+0000 7f9d0552a700 1 -- 172.21.15.169:0/2636196192 learned_addr learned my addr 172.21.15.169:0/2636196192 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:58.763 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.765+0000 7f9d0552a700 1 -- 172.21.15.169:0/2636196192 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d00101130 con 0x7f9d000703f0 2024-01-21T19:47:58.763 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.765+0000 7f9d0552a700 1 --2- 172.21.15.169:0/2636196192 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d000703f0 0x7f9d00100c40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f9cf400b5f0 tx=0x7f9cf4006ee0).ready entity=mon.0 client_cookie=85fb1453b5fcef9c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:58.763 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.765+0000 7f9cf3fff700 1 -- 172.21.15.169:0/2636196192 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cf400d600 con 0x7f9d000703f0 2024-01-21T19:47:58.763 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.765+0000 7f9cf3fff700 1 -- 172.21.15.169:0/2636196192 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9cf400dbd0 con 0x7f9d000703f0 2024-01-21T19:47:58.763 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.765+0000 7f9cf3fff700 1 -- 172.21.15.169:0/2636196192 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cf4011c30 con 0x7f9d000703f0 2024-01-21T19:47:58.763 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.766+0000 7f9d0778e700 1 -- 172.21.15.169:0/2636196192 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d000703f0 msgr2=0x7f9d00100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:58.763 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.766+0000 7f9d0778e700 1 --2- 172.21.15.169:0/2636196192 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d000703f0 0x7f9d00100c40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f9cf400b5f0 tx=0x7f9cf4006ee0).stop 2024-01-21T19:47:58.763 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.766+0000 7f9d0778e700 1 -- 172.21.15.169:0/2636196192 shutdown_connections 2024-01-21T19:47:58.763 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.766+0000 7f9d0778e700 1 --2- 172.21.15.169:0/2636196192 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d000703f0 0x7f9d00100c40 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:58.763 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.766+0000 7f9d0778e700 1 -- 172.21.15.169:0/2636196192 >> 172.21.15.169:0/2636196192 conn(0x7f9d000f62a0 msgr2=0x7f9d000f86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:58.763 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.766+0000 7f9d0778e700 1 -- 172.21.15.169:0/2636196192 shutdown_connections 2024-01-21T19:47:58.764 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.766+0000 7f9d0778e700 1 -- 172.21.15.169:0/2636196192 wait complete. 2024-01-21T19:47:58.764 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.767+0000 7f9d0778e700 1 Processor -- start 2024-01-21T19:47:58.764 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.767+0000 7f9d0778e700 1 -- start start 2024-01-21T19:47:58.764 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.767+0000 7f9d0778e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d000703f0 0x7f9d0010cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:58.764 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.767+0000 7f9d0778e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0010d0b0 con 0x7f9d000703f0 2024-01-21T19:47:58.764 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.767+0000 7f9d0552a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d000703f0 0x7f9d0010cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:58.764 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.767+0000 7f9d0552a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d000703f0 0x7f9d0010cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:37076/0 (socket says 172.21.15.169:37076) 2024-01-21T19:47:58.764 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.767+0000 7f9d0552a700 1 -- 172.21.15.169:0/3943772307 learned_addr learned my addr 172.21.15.169:0/3943772307 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:58.764 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.767+0000 7f9d0552a700 1 -- 172.21.15.169:0/3943772307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf400b200 con 0x7f9d000703f0 2024-01-21T19:47:58.764 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.767+0000 7f9d0552a700 1 --2- 172.21.15.169:0/3943772307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d000703f0 0x7f9d0010cbc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f9cf400b190 tx=0x7f9cf4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:58.764 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.767+0000 7f9cf1ffb700 1 -- 172.21.15.169:0/3943772307 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cf401b410 con 0x7f9d000703f0 2024-01-21T19:47:58.764 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.767+0000 7f9cf1ffb700 1 -- 172.21.15.169:0/3943772307 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9cf401b9e0 con 0x7f9d000703f0 2024-01-21T19:47:58.764 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.768+0000 7f9d0778e700 1 -- 172.21.15.169:0/3943772307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0010d2b0 con 0x7f9d000703f0 2024-01-21T19:47:58.765 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.768+0000 7f9cf1ffb700 1 -- 172.21.15.169:0/3943772307 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cf401a820 con 0x7f9d000703f0 2024-01-21T19:47:58.765 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.768+0000 7f9d0778e700 1 -- 172.21.15.169:0/3943772307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d0010d740 con 0x7f9d000703f0 2024-01-21T19:47:58.765 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.768+0000 7f9cf1ffb700 1 -- 172.21.15.169:0/3943772307 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f9cf400f070 con 0x7f9d000703f0 2024-01-21T19:47:58.765 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.769+0000 7f9cf1ffb700 1 --2- 172.21.15.169:0/3943772307 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f9ce802fcf0 0x7f9ce80321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:58.765 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.769+0000 7f9cf1ffb700 1 -- 172.21.15.169:0/3943772307 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f9cf4041220 con 0x7f9d000703f0 2024-01-21T19:47:58.765 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.769+0000 7f9d04d29700 1 --2- 172.21.15.169:0/3943772307 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f9ce802fcf0 0x7f9ce80321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:58.765 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.769+0000 7f9d0778e700 1 -- 172.21.15.169:0/3943772307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d00045ad0 con 0x7f9d000703f0 2024-01-21T19:47:58.765 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.769+0000 7f9d04d29700 1 --2- 172.21.15.169:0/3943772307 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f9ce802fcf0 0x7f9ce80321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9cfc0096a0 tx=0x7f9cfc006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:58.767 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.773+0000 7f9cf1ffb700 1 -- 172.21.15.169:0/3943772307 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9cf4016030 con 0x7f9d000703f0 2024-01-21T19:47:58.943 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.950+0000 7f9d0778e700 1 -- 172.21.15.169:0/3943772307 --> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d000008d0 con 0x7f9ce802fcf0 2024-01-21T19:47:58.956 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.963+0000 7f9cf1ffb700 1 -- 172.21.15.169:0/3943772307 <== mgr.14114 v2:172.21.15.169:6800/2079278379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9d000008d0 con 0x7f9ce802fcf0 2024-01-21T19:47:58.959 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.965+0000 7f9d0778e700 1 -- 172.21.15.169:0/3943772307 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f9ce802fcf0 msgr2=0x7f9ce80321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:58.959 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.965+0000 7f9d0778e700 1 --2- 172.21.15.169:0/3943772307 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f9ce802fcf0 0x7f9ce80321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9cfc0096a0 tx=0x7f9cfc006b40).stop 2024-01-21T19:47:58.959 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.965+0000 7f9d0778e700 1 -- 172.21.15.169:0/3943772307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d000703f0 msgr2=0x7f9d0010cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:58.959 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.965+0000 7f9d0778e700 1 --2- 172.21.15.169:0/3943772307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d000703f0 0x7f9d0010cbc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f9cf400b190 tx=0x7f9cf4006ee0).stop 2024-01-21T19:47:58.960 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.965+0000 7f9d0778e700 1 -- 172.21.15.169:0/3943772307 shutdown_connections 2024-01-21T19:47:58.960 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.965+0000 7f9d0778e700 1 --2- 172.21.15.169:0/3943772307 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f9ce802fcf0 0x7f9ce80321b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:58.960 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.965+0000 7f9d0778e700 1 --2- 172.21.15.169:0/3943772307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d000703f0 0x7f9d0010cbc0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:58.960 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.965+0000 7f9d0778e700 1 -- 172.21.15.169:0/3943772307 >> 172.21.15.169:0/3943772307 conn(0x7f9d000f62a0 msgr2=0x7f9d000f7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:58.960 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.965+0000 7f9d0778e700 1 -- 172.21.15.169:0/3943772307 shutdown_connections 2024-01-21T19:47:58.960 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:58.965+0000 7f9d0778e700 1 -- 172.21.15.169:0/3943772307 wait complete. 2024-01-21T19:47:59.266 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:59 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:58.018304+0000 mgr.smithi169.espgue (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-21T19:47:59.266 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:59 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:47:58.019205+0000 mon.smithi169 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi169.espgue(active, since 1.06597s) 2024-01-21T19:47:59.266 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:59 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:58.025046+0000 mgr.smithi169.espgue (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-21T19:47:59.266 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:59 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:58.951836+0000 mgr.smithi169.espgue (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:47:59.266 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:59 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:58.955767+0000 mon.smithi169 (mon.0) 49 : audit [INF] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' 2024-01-21T19:47:59.266 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:47:59 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:58.966073+0000 mon.smithi169 (mon.0) 50 : audit [DBG] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:47:59.314 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-01-21T19:47:59.672 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.675+0000 7fc65a825700 1 Processor -- start 2024-01-21T19:47:59.672 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.675+0000 7fc65a825700 1 -- start start 2024-01-21T19:47:59.672 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.675+0000 7fc65a825700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6540703f0 0x7fc6540fe790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:59.672 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.675+0000 7fc65a825700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6540fec80 con 0x7fc6540703f0 2024-01-21T19:47:59.672 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.675+0000 7fc659823700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6540703f0 0x7fc6540fe790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:59.672 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.675+0000 7fc659823700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6540703f0 0x7fc6540fe790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:42398/0 (socket says 172.21.15.169:42398) 2024-01-21T19:47:59.672 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.675+0000 7fc659823700 1 -- 172.21.15.169:0/1742356159 learned_addr learned my addr 172.21.15.169:0/1742356159 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:59.672 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.676+0000 7fc659823700 1 -- 172.21.15.169:0/1742356159 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6540fedc0 con 0x7fc6540703f0 2024-01-21T19:47:59.672 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.676+0000 7fc659823700 1 --2- 172.21.15.169:0/1742356159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6540703f0 0x7fc6540fe790 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fc65000b5f0 tx=0x7fc650006ee0).ready entity=mon.0 client_cookie=edb35d8d7ebc395c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:59.672 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.676+0000 7fc658821700 1 -- 172.21.15.169:0/1742356159 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc65000d600 con 0x7fc6540703f0 2024-01-21T19:47:59.672 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.676+0000 7fc658821700 1 -- 172.21.15.169:0/1742356159 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc65000dbd0 con 0x7fc6540703f0 2024-01-21T19:47:59.673 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.676+0000 7fc658821700 1 -- 172.21.15.169:0/1742356159 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc650011c00 con 0x7fc6540703f0 2024-01-21T19:47:59.673 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.676+0000 7fc65a825700 1 -- 172.21.15.169:0/1742356159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6540703f0 msgr2=0x7fc6540fe790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:59.673 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.676+0000 7fc65a825700 1 --2- 172.21.15.169:0/1742356159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6540703f0 0x7fc6540fe790 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fc65000b5f0 tx=0x7fc650006ee0).stop 2024-01-21T19:47:59.673 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.677+0000 7fc65a825700 1 -- 172.21.15.169:0/1742356159 shutdown_connections 2024-01-21T19:47:59.673 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.677+0000 7fc65a825700 1 --2- 172.21.15.169:0/1742356159 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6540703f0 0x7fc6540fe790 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:59.673 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.677+0000 7fc65a825700 1 -- 172.21.15.169:0/1742356159 >> 172.21.15.169:0/1742356159 conn(0x7fc6540f6210 msgr2=0x7fc6540f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:59.673 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.677+0000 7fc65a825700 1 -- 172.21.15.169:0/1742356159 shutdown_connections 2024-01-21T19:47:59.673 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.677+0000 7fc65a825700 1 -- 172.21.15.169:0/1742356159 wait complete. 2024-01-21T19:47:59.673 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.677+0000 7fc65a825700 1 Processor -- start 2024-01-21T19:47:59.673 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.677+0000 7fc65a825700 1 -- start start 2024-01-21T19:47:59.673 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.677+0000 7fc65a825700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6540703f0 0x7fc65410a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:59.673 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.677+0000 7fc65a825700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65410ad50 con 0x7fc6540703f0 2024-01-21T19:47:59.673 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.677+0000 7fc659823700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6540703f0 0x7fc65410a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:59.673 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.677+0000 7fc659823700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6540703f0 0x7fc65410a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:42408/0 (socket says 172.21.15.169:42408) 2024-01-21T19:47:59.673 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.677+0000 7fc659823700 1 -- 172.21.15.169:0/1214650497 learned_addr learned my addr 172.21.15.169:0/1214650497 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:47:59.673 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.678+0000 7fc659823700 1 -- 172.21.15.169:0/1214650497 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65000b200 con 0x7fc6540703f0 2024-01-21T19:47:59.674 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.678+0000 7fc659823700 1 --2- 172.21.15.169:0/1214650497 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6540703f0 0x7fc65410a860 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fc65000b190 tx=0x7fc650006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:59.674 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.678+0000 7fc6427fc700 1 -- 172.21.15.169:0/1214650497 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc65001b410 con 0x7fc6540703f0 2024-01-21T19:47:59.674 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.678+0000 7fc6427fc700 1 -- 172.21.15.169:0/1214650497 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc65001b9e0 con 0x7fc6540703f0 2024-01-21T19:47:59.674 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.678+0000 7fc6427fc700 1 -- 172.21.15.169:0/1214650497 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc65001a880 con 0x7fc6540703f0 2024-01-21T19:47:59.674 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.678+0000 7fc65a825700 1 -- 172.21.15.169:0/1214650497 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc65410af50 con 0x7fc6540703f0 2024-01-21T19:47:59.674 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.678+0000 7fc65a825700 1 -- 172.21.15.169:0/1214650497 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc65410b360 con 0x7fc6540703f0 2024-01-21T19:47:59.675 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.679+0000 7fc6427fc700 1 -- 172.21.15.169:0/1214650497 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7fc65000f070 con 0x7fc6540703f0 2024-01-21T19:47:59.675 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.679+0000 7fc6427fc700 1 --2- 172.21.15.169:0/1214650497 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fc63c02fcf0 0x7fc63c0321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:47:59.675 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.679+0000 7fc65a825700 1 -- 172.21.15.169:0/1214650497 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc654045ad0 con 0x7fc6540703f0 2024-01-21T19:47:59.675 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.679+0000 7fc6427fc700 1 -- 172.21.15.169:0/1214650497 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fc650041080 con 0x7fc6540703f0 2024-01-21T19:47:59.675 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.679+0000 7fc659022700 1 --2- 172.21.15.169:0/1214650497 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fc63c02fcf0 0x7fc63c0321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:47:59.675 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.679+0000 7fc659022700 1 --2- 172.21.15.169:0/1214650497 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fc63c02fcf0 0x7fc63c0321b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fc6440096a0 tx=0x7fc644006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:47:59.676 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.683+0000 7fc6427fc700 1 -- 172.21.15.169:0/1214650497 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc650016020 con 0x7fc6540703f0 2024-01-21T19:47:59.827 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.833+0000 7fc65a825700 1 -- 172.21.15.169:0/1214650497 --> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fc65410bc40 con 0x7fc63c02fcf0 2024-01-21T19:47:59.827 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.833+0000 7fc6427fc700 1 -- 172.21.15.169:0/1214650497 <== mgr.14114 v2:172.21.15.169:6800/2079278379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fc65410bc40 con 0x7fc63c02fcf0 2024-01-21T19:47:59.827 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: value unchanged 2024-01-21T19:47:59.830 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.835+0000 7fc65a825700 1 -- 172.21.15.169:0/1214650497 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fc63c02fcf0 msgr2=0x7fc63c0321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:59.830 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.835+0000 7fc65a825700 1 --2- 172.21.15.169:0/1214650497 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fc63c02fcf0 0x7fc63c0321b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fc6440096a0 tx=0x7fc644006b40).stop 2024-01-21T19:47:59.830 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.835+0000 7fc65a825700 1 -- 172.21.15.169:0/1214650497 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6540703f0 msgr2=0x7fc65410a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:47:59.830 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.835+0000 7fc65a825700 1 --2- 172.21.15.169:0/1214650497 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6540703f0 0x7fc65410a860 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fc65000b190 tx=0x7fc650006ee0).stop 2024-01-21T19:47:59.830 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.835+0000 7fc65a825700 1 -- 172.21.15.169:0/1214650497 shutdown_connections 2024-01-21T19:47:59.830 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.835+0000 7fc65a825700 1 --2- 172.21.15.169:0/1214650497 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fc63c02fcf0 0x7fc63c0321b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:59.830 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.835+0000 7fc65a825700 1 --2- 172.21.15.169:0/1214650497 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc6540703f0 0x7fc65410a860 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:47:59.830 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.835+0000 7fc65a825700 1 -- 172.21.15.169:0/1214650497 >> 172.21.15.169:0/1214650497 conn(0x7fc6540f6210 msgr2=0x7fc6540f7ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:47:59.830 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.835+0000 7fc65a825700 1 -- 172.21.15.169:0/1214650497 shutdown_connections 2024-01-21T19:47:59.830 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:47:59.835+0000 7fc65a825700 1 -- 172.21.15.169:0/1214650497 wait complete. 2024-01-21T19:48:00.180 INFO:teuthology.orchestra.run.smithi169.stderr:Generating ssh key... 2024-01-21T19:48:00.180 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-01-21T19:48:00.530 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.535+0000 7ffb36d9e700 1 Processor -- start 2024-01-21T19:48:00.530 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.535+0000 7ffb36d9e700 1 -- start start 2024-01-21T19:48:00.530 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.535+0000 7ffb36d9e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb300fa9d0 0x7ffb300fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:00.530 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.535+0000 7ffb36d9e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb300fb2e0 con 0x7ffb300fa9d0 2024-01-21T19:48:00.530 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.536+0000 7ffb35d9c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb300fa9d0 0x7ffb300fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:00.531 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.536+0000 7ffb35d9c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb300fa9d0 0x7ffb300fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:42418/0 (socket says 172.21.15.169:42418) 2024-01-21T19:48:00.531 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.536+0000 7ffb35d9c700 1 -- 172.21.15.169:0/96556610 learned_addr learned my addr 172.21.15.169:0/96556610 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:00.531 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.536+0000 7ffb35d9c700 1 -- 172.21.15.169:0/96556610 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb300fb420 con 0x7ffb300fa9d0 2024-01-21T19:48:00.531 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.536+0000 7ffb35d9c700 1 --2- 172.21.15.169:0/96556610 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb300fa9d0 0x7ffb300fadf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ffb2000b5f0 tx=0x7ffb20006ee0).ready entity=mon.0 client_cookie=3b17ca0841437fdf server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:00.531 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.536+0000 7ffb34d9a700 1 -- 172.21.15.169:0/96556610 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb2000d600 con 0x7ffb300fa9d0 2024-01-21T19:48:00.531 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.536+0000 7ffb34d9a700 1 -- 172.21.15.169:0/96556610 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ffb2000dbd0 con 0x7ffb300fa9d0 2024-01-21T19:48:00.531 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.536+0000 7ffb34d9a700 1 -- 172.21.15.169:0/96556610 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb20011c00 con 0x7ffb300fa9d0 2024-01-21T19:48:00.531 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.537+0000 7ffb36d9e700 1 -- 172.21.15.169:0/96556610 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb300fa9d0 msgr2=0x7ffb300fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:00.531 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.537+0000 7ffb36d9e700 1 --2- 172.21.15.169:0/96556610 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb300fa9d0 0x7ffb300fadf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ffb2000b5f0 tx=0x7ffb20006ee0).stop 2024-01-21T19:48:00.534 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.537+0000 7ffb36d9e700 1 -- 172.21.15.169:0/96556610 shutdown_connections 2024-01-21T19:48:00.534 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.537+0000 7ffb36d9e700 1 --2- 172.21.15.169:0/96556610 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb300fa9d0 0x7ffb300fadf0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:00.534 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.537+0000 7ffb36d9e700 1 -- 172.21.15.169:0/96556610 >> 172.21.15.169:0/96556610 conn(0x7ffb300f6210 msgr2=0x7ffb300f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:00.534 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.537+0000 7ffb36d9e700 1 -- 172.21.15.169:0/96556610 shutdown_connections 2024-01-21T19:48:00.534 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.537+0000 7ffb36d9e700 1 -- 172.21.15.169:0/96556610 wait complete. 2024-01-21T19:48:00.534 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.538+0000 7ffb36d9e700 1 Processor -- start 2024-01-21T19:48:00.534 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.538+0000 7ffb36d9e700 1 -- start start 2024-01-21T19:48:00.534 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.538+0000 7ffb36d9e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb300fa9d0 0x7ffb3010d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:00.534 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.538+0000 7ffb36d9e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb300704b0 con 0x7ffb300fa9d0 2024-01-21T19:48:00.534 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.538+0000 7ffb35d9c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb300fa9d0 0x7ffb3010d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:00.534 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.538+0000 7ffb35d9c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb300fa9d0 0x7ffb3010d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:42430/0 (socket says 172.21.15.169:42430) 2024-01-21T19:48:00.535 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.538+0000 7ffb35d9c700 1 -- 172.21.15.169:0/4027558849 learned_addr learned my addr 172.21.15.169:0/4027558849 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:00.535 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.539+0000 7ffb35d9c700 1 -- 172.21.15.169:0/4027558849 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb2000b200 con 0x7ffb300fa9d0 2024-01-21T19:48:00.535 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.539+0000 7ffb35d9c700 1 --2- 172.21.15.169:0/4027558849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb300fa9d0 0x7ffb3010d4c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7ffb2000b1d0 tx=0x7ffb20006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:00.535 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.539+0000 7ffb1e7fc700 1 -- 172.21.15.169:0/4027558849 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb2001b410 con 0x7ffb300fa9d0 2024-01-21T19:48:00.535 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.539+0000 7ffb1e7fc700 1 -- 172.21.15.169:0/4027558849 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ffb2001b9e0 con 0x7ffb300fa9d0 2024-01-21T19:48:00.535 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.539+0000 7ffb1e7fc700 1 -- 172.21.15.169:0/4027558849 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb2001a840 con 0x7ffb300fa9d0 2024-01-21T19:48:00.535 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.539+0000 7ffb36d9e700 1 -- 172.21.15.169:0/4027558849 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb300706b0 con 0x7ffb300fa9d0 2024-01-21T19:48:00.535 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.539+0000 7ffb36d9e700 1 -- 172.21.15.169:0/4027558849 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb3010db70 con 0x7ffb300fa9d0 2024-01-21T19:48:00.535 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.540+0000 7ffb1e7fc700 1 -- 172.21.15.169:0/4027558849 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ffb2000f070 con 0x7ffb300fa9d0 2024-01-21T19:48:00.535 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.540+0000 7ffb1e7fc700 1 --2- 172.21.15.169:0/4027558849 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7ffb1802fdb0 0x7ffb18032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:00.535 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.540+0000 7ffb1e7fc700 1 -- 172.21.15.169:0/4027558849 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ffb200411a0 con 0x7ffb300fa9d0 2024-01-21T19:48:00.535 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.540+0000 7ffb36d9e700 1 -- 172.21.15.169:0/4027558849 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb30045ad0 con 0x7ffb300fa9d0 2024-01-21T19:48:00.535 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.540+0000 7ffb3559b700 1 --2- 172.21.15.169:0/4027558849 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7ffb1802fdb0 0x7ffb18032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:00.535 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.540+0000 7ffb3559b700 1 --2- 172.21.15.169:0/4027558849 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7ffb1802fdb0 0x7ffb18032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ffb2c0096a0 tx=0x7ffb2c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:00.537 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.544+0000 7ffb1e7fc700 1 -- 172.21.15.169:0/4027558849 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffb20025460 con 0x7ffb300fa9d0 2024-01-21T19:48:00.689 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:00.695+0000 7ffb36d9e700 1 -- 172.21.15.169:0/4027558849 --> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb30057f10 con 0x7ffb1802fdb0 2024-01-21T19:48:01.158 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.165+0000 7ffb1e7fc700 1 -- 172.21.15.169:0/4027558849 <== mgr.14114 v2:172.21.15.169:6800/2079278379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ffb30057f10 con 0x7ffb1802fdb0 2024-01-21T19:48:01.161 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.167+0000 7ffb36d9e700 1 -- 172.21.15.169:0/4027558849 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7ffb1802fdb0 msgr2=0x7ffb18032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:01.161 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.167+0000 7ffb36d9e700 1 --2- 172.21.15.169:0/4027558849 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7ffb1802fdb0 0x7ffb18032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ffb2c0096a0 tx=0x7ffb2c006b40).stop 2024-01-21T19:48:01.161 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.167+0000 7ffb36d9e700 1 -- 172.21.15.169:0/4027558849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb300fa9d0 msgr2=0x7ffb3010d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:01.161 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.167+0000 7ffb36d9e700 1 --2- 172.21.15.169:0/4027558849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb300fa9d0 0x7ffb3010d4c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7ffb2000b1d0 tx=0x7ffb20006ee0).stop 2024-01-21T19:48:01.161 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.167+0000 7ffb36d9e700 1 -- 172.21.15.169:0/4027558849 shutdown_connections 2024-01-21T19:48:01.161 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.167+0000 7ffb36d9e700 1 --2- 172.21.15.169:0/4027558849 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7ffb1802fdb0 0x7ffb18032270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:01.161 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.167+0000 7ffb36d9e700 1 --2- 172.21.15.169:0/4027558849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb300fa9d0 0x7ffb3010d4c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:01.161 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.167+0000 7ffb36d9e700 1 -- 172.21.15.169:0/4027558849 >> 172.21.15.169:0/4027558849 conn(0x7ffb300f6210 msgr2=0x7ffb300f7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:01.161 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.167+0000 7ffb36d9e700 1 -- 172.21.15.169:0/4027558849 shutdown_connections 2024-01-21T19:48:01.161 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.168+0000 7ffb36d9e700 1 -- 172.21.15.169:0/4027558849 wait complete. 2024-01-21T19:48:01.339 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:00 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:47:59.834659+0000 mgr.smithi169.espgue (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:48:01.339 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:00 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:47:59.956576+0000 mon.smithi169 (mon.0) 51 : cluster [DBG] mgrmap e8: smithi169.espgue(active, since 3s) 2024-01-21T19:48:01.505 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-01-21T19:48:01.861 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.865+0000 7fe08cfb6700 1 Processor -- start 2024-01-21T19:48:01.861 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.865+0000 7fe08cfb6700 1 -- start start 2024-01-21T19:48:01.861 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.866+0000 7fe08cfb6700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe088070430 0x7fe0880fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:01.861 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.866+0000 7fe08cfb6700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0880fedd0 con 0x7fe088070430 2024-01-21T19:48:01.861 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.866+0000 7fe08659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe088070430 0x7fe0880fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:01.861 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.866+0000 7fe08659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe088070430 0x7fe0880fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:42440/0 (socket says 172.21.15.169:42440) 2024-01-21T19:48:01.861 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.866+0000 7fe08659c700 1 -- 172.21.15.169:0/1473293516 learned_addr learned my addr 172.21.15.169:0/1473293516 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:01.861 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.866+0000 7fe08659c700 1 -- 172.21.15.169:0/1473293516 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0880fef10 con 0x7fe088070430 2024-01-21T19:48:01.861 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.866+0000 7fe08659c700 1 --2- 172.21.15.169:0/1473293516 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe088070430 0x7fe0880fe8e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fe07000b5f0 tx=0x7fe070006ee0).ready entity=mon.0 client_cookie=881315ce99af5050 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:01.862 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.867+0000 7fe08559a700 1 -- 172.21.15.169:0/1473293516 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe07000d600 con 0x7fe088070430 2024-01-21T19:48:01.862 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.867+0000 7fe08559a700 1 -- 172.21.15.169:0/1473293516 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe07000dbd0 con 0x7fe088070430 2024-01-21T19:48:01.862 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.867+0000 7fe08559a700 1 -- 172.21.15.169:0/1473293516 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe070011c00 con 0x7fe088070430 2024-01-21T19:48:01.862 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.867+0000 7fe08cfb6700 1 -- 172.21.15.169:0/1473293516 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe088070430 msgr2=0x7fe0880fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:01.862 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.867+0000 7fe08cfb6700 1 --2- 172.21.15.169:0/1473293516 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe088070430 0x7fe0880fe8e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fe07000b5f0 tx=0x7fe070006ee0).stop 2024-01-21T19:48:01.862 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.867+0000 7fe08cfb6700 1 -- 172.21.15.169:0/1473293516 shutdown_connections 2024-01-21T19:48:01.862 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.867+0000 7fe08cfb6700 1 --2- 172.21.15.169:0/1473293516 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe088070430 0x7fe0880fe8e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:01.862 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.867+0000 7fe08cfb6700 1 -- 172.21.15.169:0/1473293516 >> 172.21.15.169:0/1473293516 conn(0x7fe0880f63c0 msgr2=0x7fe0880f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:01.862 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.867+0000 7fe08cfb6700 1 -- 172.21.15.169:0/1473293516 shutdown_connections 2024-01-21T19:48:01.862 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.867+0000 7fe08cfb6700 1 -- 172.21.15.169:0/1473293516 wait complete. 2024-01-21T19:48:01.864 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.868+0000 7fe08cfb6700 1 Processor -- start 2024-01-21T19:48:01.864 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.868+0000 7fe08cfb6700 1 -- start start 2024-01-21T19:48:01.864 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.868+0000 7fe08cfb6700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe088070430 0x7fe0880fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:01.865 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.868+0000 7fe08cfb6700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0880fa7e0 con 0x7fe088070430 2024-01-21T19:48:01.865 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.868+0000 7fe08659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe088070430 0x7fe0880fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:01.865 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.868+0000 7fe08659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe088070430 0x7fe0880fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:42444/0 (socket says 172.21.15.169:42444) 2024-01-21T19:48:01.865 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.868+0000 7fe08659c700 1 -- 172.21.15.169:0/95746512 learned_addr learned my addr 172.21.15.169:0/95746512 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:01.865 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.869+0000 7fe08659c700 1 -- 172.21.15.169:0/95746512 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07000b200 con 0x7fe088070430 2024-01-21T19:48:01.865 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.869+0000 7fe08659c700 1 --2- 172.21.15.169:0/95746512 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe088070430 0x7fe0880fc360 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fe07000b190 tx=0x7fe070006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:01.865 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.869+0000 7fe07effd700 1 -- 172.21.15.169:0/95746512 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe07001b410 con 0x7fe088070430 2024-01-21T19:48:01.865 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.869+0000 7fe07effd700 1 -- 172.21.15.169:0/95746512 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe07001b9e0 con 0x7fe088070430 2024-01-21T19:48:01.865 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.869+0000 7fe08cfb6700 1 -- 172.21.15.169:0/95746512 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0880fa980 con 0x7fe088070430 2024-01-21T19:48:01.865 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.869+0000 7fe07effd700 1 -- 172.21.15.169:0/95746512 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe07001a880 con 0x7fe088070430 2024-01-21T19:48:01.865 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.869+0000 7fe08cfb6700 1 -- 172.21.15.169:0/95746512 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0880fad90 con 0x7fe088070430 2024-01-21T19:48:01.865 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.870+0000 7fe08cfb6700 1 -- 172.21.15.169:0/95746512 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe088045ad0 con 0x7fe088070430 2024-01-21T19:48:01.865 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.870+0000 7fe07effd700 1 -- 172.21.15.169:0/95746512 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fe07000f070 con 0x7fe088070430 2024-01-21T19:48:01.865 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.870+0000 7fe07effd700 1 --2- 172.21.15.169:0/95746512 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fe06802fd50 0x7fe068032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:01.866 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.870+0000 7fe07effd700 1 -- 172.21.15.169:0/95746512 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fe070041290 con 0x7fe088070430 2024-01-21T19:48:01.866 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.870+0000 7fe085d9b700 1 --2- 172.21.15.169:0/95746512 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fe06802fd50 0x7fe068032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:01.866 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.871+0000 7fe085d9b700 1 --2- 172.21.15.169:0/95746512 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fe06802fd50 0x7fe068032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fe0780096a0 tx=0x7fe078006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:01.867 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:01.874+0000 7fe07effd700 1 -- 172.21.15.169:0/95746512 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe070019cd0 con 0x7fe088070430 2024-01-21T19:48:02.022 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.028+0000 7fe08cfb6700 1 -- 172.21.15.169:0/95746512 --> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0880fafd0 con 0x7fe06802fd50 2024-01-21T19:48:02.022 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.028+0000 7fe07effd700 1 -- 172.21.15.169:0/95746512 <== mgr.14114 v2:172.21.15.169:6800/2079278379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fe0880fafd0 con 0x7fe06802fd50 2024-01-21T19:48:02.022 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCietmYFToaJmPcbdPHcOmy9X42250DP3qHK5+K+bMtLXnyGzcLoanwMtoeJCt0tGfULcTaa0jrynfe9jvQXV79ZOQsacdy6wlsmtCG1TukjkgBYE/DpdZuHDtLgJneMGI7IfNcRIosumvWqAz7ENYIn1s+fARbd1DSoJE7JdAFroqwH7bLatsjd8gs8F/BFzHvwJjzSu0ayEfvI45oZU7PxybCBFPRh8gbVhKmxZn9IAw0GwuBw4aiN72JLymqq5zxWMy6CuMw9g0GrHTbQfoXyw/87oitQr9tk9Auvfm/ohRrhSmUC8oAvdUJ7XIXQxV2qrlAZHXn67weC6yc4GzjHDAl/gOIy0mnULHkEie/+Uj7ZjablRtP9dzkxIPAE2Uq9SKChWAZ7mfHebA7DBD1Kdhja5GaOZ7cAE1WknlNT4SaVu7kfwkq0cju130bsqzxQCGaueWAgfu1aSv9nydEt+6z7NA1Cmi8DzApO3Z6Z++WpQ5bFTkFUaSQZyDEc1k= ceph-df2470f2-b895-11ee-95b0-87774f69a715 2024-01-21T19:48:02.024 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.030+0000 7fe08cfb6700 1 -- 172.21.15.169:0/95746512 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fe06802fd50 msgr2=0x7fe068032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:02.024 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.030+0000 7fe08cfb6700 1 --2- 172.21.15.169:0/95746512 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fe06802fd50 0x7fe068032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fe0780096a0 tx=0x7fe078006b40).stop 2024-01-21T19:48:02.024 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.030+0000 7fe08cfb6700 1 -- 172.21.15.169:0/95746512 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe088070430 msgr2=0x7fe0880fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:02.024 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.030+0000 7fe08cfb6700 1 --2- 172.21.15.169:0/95746512 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe088070430 0x7fe0880fc360 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fe07000b190 tx=0x7fe070006ee0).stop 2024-01-21T19:48:02.024 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.030+0000 7fe08cfb6700 1 -- 172.21.15.169:0/95746512 shutdown_connections 2024-01-21T19:48:02.025 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.030+0000 7fe08cfb6700 1 --2- 172.21.15.169:0/95746512 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fe06802fd50 0x7fe068032210 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:02.025 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.030+0000 7fe08cfb6700 1 --2- 172.21.15.169:0/95746512 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe088070430 0x7fe0880fc360 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:02.025 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.030+0000 7fe08cfb6700 1 -- 172.21.15.169:0/95746512 >> 172.21.15.169:0/95746512 conn(0x7fe0880f63c0 msgr2=0x7fe0880f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:02.025 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.030+0000 7fe08cfb6700 1 -- 172.21.15.169:0/95746512 shutdown_connections 2024-01-21T19:48:02.025 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.030+0000 7fe08cfb6700 1 -- 172.21.15.169:0/95746512 wait complete. 2024-01-21T19:48:02.315 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:01 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cephadm 2024-01-21T19:48:00.697438+0000 mgr.smithi169.espgue (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-01-21T19:48:02.315 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:01 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:01.162780+0000 mon.smithi169 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' 2024-01-21T19:48:02.316 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:01 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:01.165173+0000 mon.smithi169 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' 2024-01-21T19:48:02.371 INFO:teuthology.orchestra.run.smithi169.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-21T19:48:02.372 INFO:teuthology.orchestra.run.smithi169.stderr:Adding key to root@localhost authorized_keys... 2024-01-21T19:48:02.372 INFO:teuthology.orchestra.run.smithi169.stderr:Adding host smithi169... 2024-01-21T19:48:02.372 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi169 172.21.15.169 2024-01-21T19:48:02.752 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.754+0000 7fd1d20e3700 1 Processor -- start 2024-01-21T19:48:02.752 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.755+0000 7fd1d20e3700 1 -- start start 2024-01-21T19:48:02.752 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.755+0000 7fd1d20e3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1cc070430 0x7fd1cc0fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:02.753 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.755+0000 7fd1d20e3700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc0fec20 con 0x7fd1cc070430 2024-01-21T19:48:02.753 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.755+0000 7fd1d10e1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1cc070430 0x7fd1cc0fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:02.753 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.755+0000 7fd1d10e1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1cc070430 0x7fd1cc0fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:42448/0 (socket says 172.21.15.169:42448) 2024-01-21T19:48:02.753 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.755+0000 7fd1d10e1700 1 -- 172.21.15.169:0/424667670 learned_addr learned my addr 172.21.15.169:0/424667670 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:02.753 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.755+0000 7fd1d10e1700 1 -- 172.21.15.169:0/424667670 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1cc0fed60 con 0x7fd1cc070430 2024-01-21T19:48:02.753 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.755+0000 7fd1d10e1700 1 --2- 172.21.15.169:0/424667670 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1cc070430 0x7fd1cc0fe730 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fd1bc00b5f0 tx=0x7fd1bc006ee0).ready entity=mon.0 client_cookie=1aa51cadb167b148 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:02.753 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.756+0000 7fd1bbfff700 1 -- 172.21.15.169:0/424667670 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1bc00d600 con 0x7fd1cc070430 2024-01-21T19:48:02.753 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.756+0000 7fd1bbfff700 1 -- 172.21.15.169:0/424667670 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd1bc00dbd0 con 0x7fd1cc070430 2024-01-21T19:48:02.753 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.756+0000 7fd1bbfff700 1 -- 172.21.15.169:0/424667670 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1bc011c00 con 0x7fd1cc070430 2024-01-21T19:48:02.753 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.756+0000 7fd1d20e3700 1 -- 172.21.15.169:0/424667670 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1cc070430 msgr2=0x7fd1cc0fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:02.753 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.756+0000 7fd1d20e3700 1 --2- 172.21.15.169:0/424667670 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1cc070430 0x7fd1cc0fe730 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fd1bc00b5f0 tx=0x7fd1bc006ee0).stop 2024-01-21T19:48:02.753 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.756+0000 7fd1d20e3700 1 -- 172.21.15.169:0/424667670 shutdown_connections 2024-01-21T19:48:02.753 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.756+0000 7fd1d20e3700 1 --2- 172.21.15.169:0/424667670 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1cc070430 0x7fd1cc0fe730 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.756+0000 7fd1d20e3700 1 -- 172.21.15.169:0/424667670 >> 172.21.15.169:0/424667670 conn(0x7fd1cc0f6210 msgr2=0x7fd1cc0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.756+0000 7fd1d20e3700 1 -- 172.21.15.169:0/424667670 shutdown_connections 2024-01-21T19:48:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.757+0000 7fd1d20e3700 1 -- 172.21.15.169:0/424667670 wait complete. 2024-01-21T19:48:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.757+0000 7fd1d20e3700 1 Processor -- start 2024-01-21T19:48:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.757+0000 7fd1d20e3700 1 -- start start 2024-01-21T19:48:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.757+0000 7fd1d20e3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1cc070430 0x7fd1cc10a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.757+0000 7fd1d20e3700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc10ad20 con 0x7fd1cc070430 2024-01-21T19:48:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.757+0000 7fd1d10e1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1cc070430 0x7fd1cc10a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.757+0000 7fd1d10e1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1cc070430 0x7fd1cc10a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:42456/0 (socket says 172.21.15.169:42456) 2024-01-21T19:48:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.757+0000 7fd1d10e1700 1 -- 172.21.15.169:0/3899220734 learned_addr learned my addr 172.21.15.169:0/3899220734 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.758+0000 7fd1d10e1700 1 -- 172.21.15.169:0/3899220734 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc00b200 con 0x7fd1cc070430 2024-01-21T19:48:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.758+0000 7fd1d10e1700 1 --2- 172.21.15.169:0/3899220734 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1cc070430 0x7fd1cc10a830 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fd1bc00b190 tx=0x7fd1bc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.758+0000 7fd1b9ffb700 1 -- 172.21.15.169:0/3899220734 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1bc01b410 con 0x7fd1cc070430 2024-01-21T19:48:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.758+0000 7fd1b9ffb700 1 -- 172.21.15.169:0/3899220734 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd1bc01b9e0 con 0x7fd1cc070430 2024-01-21T19:48:02.755 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.758+0000 7fd1d20e3700 1 -- 172.21.15.169:0/3899220734 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1cc10af20 con 0x7fd1cc070430 2024-01-21T19:48:02.755 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.758+0000 7fd1d20e3700 1 -- 172.21.15.169:0/3899220734 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1cc10b330 con 0x7fd1cc070430 2024-01-21T19:48:02.755 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.758+0000 7fd1b9ffb700 1 -- 172.21.15.169:0/3899220734 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1bc01a800 con 0x7fd1cc070430 2024-01-21T19:48:02.755 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.759+0000 7fd1d20e3700 1 -- 172.21.15.169:0/3899220734 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1cc045ad0 con 0x7fd1cc070430 2024-01-21T19:48:02.756 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.759+0000 7fd1b9ffb700 1 -- 172.21.15.169:0/3899220734 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fd1bc00f070 con 0x7fd1cc070430 2024-01-21T19:48:02.756 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.759+0000 7fd1b9ffb700 1 --2- 172.21.15.169:0/3899220734 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fd1b402fd50 0x7fd1b4032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:02.756 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.759+0000 7fd1b9ffb700 1 -- 172.21.15.169:0/3899220734 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fd1bc0418e0 con 0x7fd1cc070430 2024-01-21T19:48:02.756 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.760+0000 7fd1d08e0700 1 --2- 172.21.15.169:0/3899220734 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fd1b402fd50 0x7fd1b4032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:02.756 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.760+0000 7fd1d08e0700 1 --2- 172.21.15.169:0/3899220734 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fd1b402fd50 0x7fd1b4032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fd1c80096a0 tx=0x7fd1c8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:02.756 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.763+0000 7fd1b9ffb700 1 -- 172.21.15.169:0/3899220734 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd1bc02dd70 con 0x7fd1cc070430 2024-01-21T19:48:02.931 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:02.938+0000 7fd1d20e3700 1 -- 172.21.15.169:0/3899220734 --> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi169", "addr": "172.21.15.169", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1cc10bc10 con 0x7fd1b402fd50 2024-01-21T19:48:03.559 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:03.564+0000 7fd1b9ffb700 1 -- 172.21.15.169:0/3899220734 <== mgr.14114 v2:172.21.15.169:6800/2079278379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fd1cc10bc10 con 0x7fd1b402fd50 2024-01-21T19:48:03.559 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: Added host 'smithi169' 2024-01-21T19:48:03.560 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:03.567+0000 7fd1d20e3700 1 -- 172.21.15.169:0/3899220734 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fd1b402fd50 msgr2=0x7fd1b4032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:03.560 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:03.567+0000 7fd1d20e3700 1 --2- 172.21.15.169:0/3899220734 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fd1b402fd50 0x7fd1b4032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fd1c80096a0 tx=0x7fd1c8006b40).stop 2024-01-21T19:48:03.561 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:03.567+0000 7fd1d20e3700 1 -- 172.21.15.169:0/3899220734 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1cc070430 msgr2=0x7fd1cc10a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:03.561 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:03.567+0000 7fd1d20e3700 1 --2- 172.21.15.169:0/3899220734 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1cc070430 0x7fd1cc10a830 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fd1bc00b190 tx=0x7fd1bc006ee0).stop 2024-01-21T19:48:03.561 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:03.568+0000 7fd1d20e3700 1 -- 172.21.15.169:0/3899220734 shutdown_connections 2024-01-21T19:48:03.562 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:03.568+0000 7fd1d20e3700 1 --2- 172.21.15.169:0/3899220734 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fd1b402fd50 0x7fd1b4032210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:03.562 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:03.568+0000 7fd1d20e3700 1 --2- 172.21.15.169:0/3899220734 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1cc070430 0x7fd1cc10a830 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:03.562 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:03.568+0000 7fd1d20e3700 1 -- 172.21.15.169:0/3899220734 >> 172.21.15.169:0/3899220734 conn(0x7fd1cc0f6210 msgr2=0x7fd1cc0f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:03.562 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:03.568+0000 7fd1d20e3700 1 -- 172.21.15.169:0/3899220734 shutdown_connections 2024-01-21T19:48:03.562 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:03.568+0000 7fd1d20e3700 1 -- 172.21.15.169:0/3899220734 wait complete. 2024-01-21T19:48:03.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:03 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:02.029561+0000 mgr.smithi169.espgue (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:48:03.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:03 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:02.939915+0000 mgr.smithi169.espgue (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi169", "addr": "172.21.15.169", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:48:03.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:03 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:03.565103+0000 mon.smithi169 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' 2024-01-21T19:48:03.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:03 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:03.565647+0000 mon.smithi169 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:48:04.917 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:04 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cephadm 2024-01-21T19:48:03.565403+0000 mgr.smithi169.espgue (mgr.14114) 9 : cephadm [INF] Added host smithi169 2024-01-21T19:48:05.305 INFO:teuthology.orchestra.run.smithi169.stderr:Deploying mon service with default placement... 2024-01-21T19:48:05.305 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-01-21T19:48:06.566 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.523+0000 7f19ba946700 1 Processor -- start 2024-01-21T19:48:06.567 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.523+0000 7f19ba946700 1 -- start start 2024-01-21T19:48:06.567 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.523+0000 7f19ba946700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b4072500 0x7f19b4070aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:06.567 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.523+0000 7f19ba946700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b4072920 con 0x7f19b4072500 2024-01-21T19:48:06.567 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.524+0000 7f19b3fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b4072500 0x7f19b4070aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:06.567 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.524+0000 7f19b3fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b4072500 0x7f19b4070aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:42462/0 (socket says 172.21.15.169:42462) 2024-01-21T19:48:06.567 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.524+0000 7f19b3fff700 1 -- 172.21.15.169:0/86115901 learned_addr learned my addr 172.21.15.169:0/86115901 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:06.567 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.524+0000 7f19b3fff700 1 -- 172.21.15.169:0/86115901 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b4070f90 con 0x7f19b4072500 2024-01-21T19:48:06.567 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.524+0000 7f19b3fff700 1 --2- 172.21.15.169:0/86115901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b4072500 0x7f19b4070aa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f19a400b5f0 tx=0x7f19a4006ee0).ready entity=mon.0 client_cookie=1a776ec7443678f8 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:06.567 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.524+0000 7f19b37fe700 1 -- 172.21.15.169:0/86115901 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19a400d600 con 0x7f19b4072500 2024-01-21T19:48:06.567 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.524+0000 7f19b37fe700 1 -- 172.21.15.169:0/86115901 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f19a400dbd0 con 0x7f19b4072500 2024-01-21T19:48:06.567 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.524+0000 7f19b37fe700 1 -- 172.21.15.169:0/86115901 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19a4011c00 con 0x7f19b4072500 2024-01-21T19:48:06.567 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.525+0000 7f19ba946700 1 -- 172.21.15.169:0/86115901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b4072500 msgr2=0x7f19b4070aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:06.568 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.525+0000 7f19ba946700 1 --2- 172.21.15.169:0/86115901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b4072500 0x7f19b4070aa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f19a400b5f0 tx=0x7f19a4006ee0).stop 2024-01-21T19:48:06.568 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.525+0000 7f19ba946700 1 -- 172.21.15.169:0/86115901 shutdown_connections 2024-01-21T19:48:06.568 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.525+0000 7f19ba946700 1 --2- 172.21.15.169:0/86115901 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b4072500 0x7f19b4070aa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:06.568 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.525+0000 7f19ba946700 1 -- 172.21.15.169:0/86115901 >> 172.21.15.169:0/86115901 conn(0x7f19b40f74b0 msgr2=0x7f19b40f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:06.568 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.525+0000 7f19ba946700 1 -- 172.21.15.169:0/86115901 shutdown_connections 2024-01-21T19:48:06.568 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.525+0000 7f19ba946700 1 -- 172.21.15.169:0/86115901 wait complete. 2024-01-21T19:48:06.568 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.526+0000 7f19ba946700 1 Processor -- start 2024-01-21T19:48:06.568 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.526+0000 7f19ba946700 1 -- start start 2024-01-21T19:48:06.568 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.526+0000 7f19ba946700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b4072500 0x7f19b410cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:06.568 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.526+0000 7f19ba946700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b410d1f0 con 0x7f19b4072500 2024-01-21T19:48:06.568 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.526+0000 7f19b3fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b4072500 0x7f19b410cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:06.568 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.526+0000 7f19b3fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b4072500 0x7f19b410cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:42478/0 (socket says 172.21.15.169:42478) 2024-01-21T19:48:06.568 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.526+0000 7f19b3fff700 1 -- 172.21.15.169:0/3361171322 learned_addr learned my addr 172.21.15.169:0/3361171322 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:06.568 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.526+0000 7f19b3fff700 1 -- 172.21.15.169:0/3361171322 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a400b200 con 0x7f19b4072500 2024-01-21T19:48:06.569 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.527+0000 7f19b3fff700 1 --2- 172.21.15.169:0/3361171322 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b4072500 0x7f19b410cd00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f19a4011820 tx=0x7f19a4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:06.569 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.527+0000 7f19b17fa700 1 -- 172.21.15.169:0/3361171322 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19a401b410 con 0x7f19b4072500 2024-01-21T19:48:06.569 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.527+0000 7f19b17fa700 1 -- 172.21.15.169:0/3361171322 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f19a401b9e0 con 0x7f19b4072500 2024-01-21T19:48:06.569 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.527+0000 7f19ba946700 1 -- 172.21.15.169:0/3361171322 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b410d3f0 con 0x7f19b4072500 2024-01-21T19:48:06.569 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.527+0000 7f19b17fa700 1 -- 172.21.15.169:0/3361171322 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19a401a8b0 con 0x7f19b4072500 2024-01-21T19:48:06.569 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.527+0000 7f19ba946700 1 -- 172.21.15.169:0/3361171322 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b410d800 con 0x7f19b4072500 2024-01-21T19:48:06.569 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.528+0000 7f19b17fa700 1 -- 172.21.15.169:0/3361171322 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f19a400f070 con 0x7f19b4072500 2024-01-21T19:48:06.569 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.528+0000 7f19b17fa700 1 --2- 172.21.15.169:0/3361171322 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f199402fdb0 0x7f1994032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:06.569 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.528+0000 7f19ba946700 1 -- 172.21.15.169:0/3361171322 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19b4045ad0 con 0x7f19b4072500 2024-01-21T19:48:06.569 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.528+0000 7f19b17fa700 1 -- 172.21.15.169:0/3361171322 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f19a40412e0 con 0x7f19b4072500 2024-01-21T19:48:06.569 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.528+0000 7f19abfff700 1 --2- 172.21.15.169:0/3361171322 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f199402fdb0 0x7f1994032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:06.569 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.528+0000 7f19abfff700 1 --2- 172.21.15.169:0/3361171322 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f199402fdb0 0x7f1994032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f199c0096a0 tx=0x7f199c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:06.569 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.532+0000 7f19b17fa700 1 -- 172.21.15.169:0/3361171322 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f19a4016070 con 0x7f19b4072500 2024-01-21T19:48:06.703 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.710+0000 7f19ba946700 1 -- 172.21.15.169:0/3361171322 --> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f19b410df30 con 0x7f199402fdb0 2024-01-21T19:48:06.707 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.714+0000 7f19b17fa700 1 -- 172.21.15.169:0/3361171322 <== mgr.14114 v2:172.21.15.169:6800/2079278379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f19b410df30 con 0x7f199402fdb0 2024-01-21T19:48:06.707 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: Scheduled mon update... 2024-01-21T19:48:06.710 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.716+0000 7f19ba946700 1 -- 172.21.15.169:0/3361171322 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f199402fdb0 msgr2=0x7f1994032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:06.710 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.716+0000 7f19ba946700 1 --2- 172.21.15.169:0/3361171322 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f199402fdb0 0x7f1994032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f199c0096a0 tx=0x7f199c006b40).stop 2024-01-21T19:48:06.710 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.716+0000 7f19ba946700 1 -- 172.21.15.169:0/3361171322 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b4072500 msgr2=0x7f19b410cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:06.710 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.716+0000 7f19ba946700 1 --2- 172.21.15.169:0/3361171322 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b4072500 0x7f19b410cd00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f19a4011820 tx=0x7f19a4006ee0).stop 2024-01-21T19:48:06.710 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.716+0000 7f19ba946700 1 -- 172.21.15.169:0/3361171322 shutdown_connections 2024-01-21T19:48:06.710 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.716+0000 7f19ba946700 1 --2- 172.21.15.169:0/3361171322 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f199402fdb0 0x7f1994032270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:06.710 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.716+0000 7f19ba946700 1 --2- 172.21.15.169:0/3361171322 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f19b4072500 0x7f19b410cd00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:06.710 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.716+0000 7f19ba946700 1 -- 172.21.15.169:0/3361171322 >> 172.21.15.169:0/3361171322 conn(0x7f19b40f74b0 msgr2=0x7f19b40f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:06.711 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.716+0000 7f19ba946700 1 -- 172.21.15.169:0/3361171322 shutdown_connections 2024-01-21T19:48:06.711 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:06.716+0000 7f19ba946700 1 -- 172.21.15.169:0/3361171322 wait complete. 2024-01-21T19:48:08.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:07 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:06.711988+0000 mgr.smithi169.espgue (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:48:08.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:07 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cephadm 2024-01-21T19:48:06.712702+0000 mgr.smithi169.espgue (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-01-21T19:48:08.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:07 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:06.714962+0000 mon.smithi169 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' 2024-01-21T19:48:08.604 INFO:teuthology.orchestra.run.smithi169.stderr:Deploying mgr service with default placement... 2024-01-21T19:48:08.604 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-01-21T19:48:09.952 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.958+0000 7fabce9ca700 1 Processor -- start 2024-01-21T19:48:09.954 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.958+0000 7fabce9ca700 1 -- start start 2024-01-21T19:48:09.954 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.958+0000 7fabce9ca700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabc80703f0 0x7fabc80fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:09.954 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.958+0000 7fabce9ca700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc80fee10 con 0x7fabc80703f0 2024-01-21T19:48:09.954 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.959+0000 7fabc7fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabc80703f0 0x7fabc80fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:09.955 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.959+0000 7fabc7fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabc80703f0 0x7fabc80fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49768/0 (socket says 172.21.15.169:49768) 2024-01-21T19:48:09.955 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.959+0000 7fabc7fff700 1 -- 172.21.15.169:0/3949175415 learned_addr learned my addr 172.21.15.169:0/3949175415 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:09.955 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.959+0000 7fabc7fff700 1 -- 172.21.15.169:0/3949175415 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc80fef50 con 0x7fabc80703f0 2024-01-21T19:48:09.955 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.959+0000 7fabc7fff700 1 --2- 172.21.15.169:0/3949175415 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabc80703f0 0x7fabc80fe920 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fabb800b5f0 tx=0x7fabb8006ee0).ready entity=mon.0 client_cookie=52bce6559479b0ae server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:09.955 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.959+0000 7fabc6ffd700 1 -- 172.21.15.169:0/3949175415 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabb800d600 con 0x7fabc80703f0 2024-01-21T19:48:09.955 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.959+0000 7fabc6ffd700 1 -- 172.21.15.169:0/3949175415 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fabb800dbd0 con 0x7fabc80703f0 2024-01-21T19:48:09.955 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.959+0000 7fabc6ffd700 1 -- 172.21.15.169:0/3949175415 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabb8011c00 con 0x7fabc80703f0 2024-01-21T19:48:09.955 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.960+0000 7fabce9ca700 1 -- 172.21.15.169:0/3949175415 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabc80703f0 msgr2=0x7fabc80fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:09.955 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.960+0000 7fabce9ca700 1 --2- 172.21.15.169:0/3949175415 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabc80703f0 0x7fabc80fe920 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fabb800b5f0 tx=0x7fabb8006ee0).stop 2024-01-21T19:48:09.955 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.960+0000 7fabce9ca700 1 -- 172.21.15.169:0/3949175415 shutdown_connections 2024-01-21T19:48:09.955 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.960+0000 7fabce9ca700 1 --2- 172.21.15.169:0/3949175415 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabc80703f0 0x7fabc80fe920 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:09.955 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.960+0000 7fabce9ca700 1 -- 172.21.15.169:0/3949175415 >> 172.21.15.169:0/3949175415 conn(0x7fabc80f63c0 msgr2=0x7fabc80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:09.955 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.960+0000 7fabce9ca700 1 -- 172.21.15.169:0/3949175415 shutdown_connections 2024-01-21T19:48:09.955 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.960+0000 7fabce9ca700 1 -- 172.21.15.169:0/3949175415 wait complete. 2024-01-21T19:48:09.955 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.960+0000 7fabce9ca700 1 Processor -- start 2024-01-21T19:48:09.956 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.961+0000 7fabce9ca700 1 -- start start 2024-01-21T19:48:09.957 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.961+0000 7fabce9ca700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabc80703f0 0x7fabc80fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:09.958 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.961+0000 7fabce9ca700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc80fa820 con 0x7fabc80703f0 2024-01-21T19:48:09.958 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.961+0000 7fabc7fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabc80703f0 0x7fabc80fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:09.958 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.961+0000 7fabc7fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabc80703f0 0x7fabc80fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49778/0 (socket says 172.21.15.169:49778) 2024-01-21T19:48:09.958 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.961+0000 7fabc7fff700 1 -- 172.21.15.169:0/659932888 learned_addr learned my addr 172.21.15.169:0/659932888 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:09.958 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.961+0000 7fabc7fff700 1 -- 172.21.15.169:0/659932888 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb800b200 con 0x7fabc80703f0 2024-01-21T19:48:09.958 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.961+0000 7fabc7fff700 1 --2- 172.21.15.169:0/659932888 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabc80703f0 0x7fabc80fc3a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fabb800e040 tx=0x7fabb8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:09.958 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.962+0000 7fabc4ff9700 1 -- 172.21.15.169:0/659932888 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabb800d9f0 con 0x7fabc80703f0 2024-01-21T19:48:09.958 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.962+0000 7fabc4ff9700 1 -- 172.21.15.169:0/659932888 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fabb801a9f0 con 0x7fabc80703f0 2024-01-21T19:48:09.958 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.962+0000 7fabce9ca700 1 -- 172.21.15.169:0/659932888 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc80fa9c0 con 0x7fabc80703f0 2024-01-21T19:48:09.958 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.962+0000 7fabc4ff9700 1 -- 172.21.15.169:0/659932888 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabb801b8c0 con 0x7fabc80703f0 2024-01-21T19:48:09.958 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.962+0000 7fabce9ca700 1 -- 172.21.15.169:0/659932888 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc80fadd0 con 0x7fabc80703f0 2024-01-21T19:48:09.958 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.963+0000 7fabce9ca700 1 -- 172.21.15.169:0/659932888 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc8045ad0 con 0x7fabc80703f0 2024-01-21T19:48:09.958 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.963+0000 7fabc4ff9700 1 -- 172.21.15.169:0/659932888 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fabb800f070 con 0x7fabc80703f0 2024-01-21T19:48:09.958 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.963+0000 7fabc4ff9700 1 --2- 172.21.15.169:0/659932888 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7faba802fd50 0x7faba8032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:09.958 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.963+0000 7fabc4ff9700 1 -- 172.21.15.169:0/659932888 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fabb8041320 con 0x7fabc80703f0 2024-01-21T19:48:09.959 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.963+0000 7fabc77fe700 1 --2- 172.21.15.169:0/659932888 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7faba802fd50 0x7faba8032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:09.959 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.963+0000 7fabc77fe700 1 --2- 172.21.15.169:0/659932888 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7faba802fd50 0x7faba8032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fabb00096a0 tx=0x7fabb0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:09.960 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:09.967+0000 7fabc4ff9700 1 -- 172.21.15.169:0/659932888 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fabb80224f0 con 0x7fabc80703f0 2024-01-21T19:48:10.136 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.142+0000 7fabce9ca700 1 -- 172.21.15.169:0/659932888 --> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fabc80ffd50 con 0x7faba802fd50 2024-01-21T19:48:10.140 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.146+0000 7fabc4ff9700 1 -- 172.21.15.169:0/659932888 <== mgr.14114 v2:172.21.15.169:6800/2079278379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fabc80ffd50 con 0x7faba802fd50 2024-01-21T19:48:10.140 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-01-21T19:48:10.142 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.148+0000 7fabce9ca700 1 -- 172.21.15.169:0/659932888 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7faba802fd50 msgr2=0x7faba8032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:10.143 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.148+0000 7fabce9ca700 1 --2- 172.21.15.169:0/659932888 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7faba802fd50 0x7faba8032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fabb00096a0 tx=0x7fabb0006b40).stop 2024-01-21T19:48:10.143 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.148+0000 7fabce9ca700 1 -- 172.21.15.169:0/659932888 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabc80703f0 msgr2=0x7fabc80fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:10.143 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.148+0000 7fabce9ca700 1 --2- 172.21.15.169:0/659932888 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabc80703f0 0x7fabc80fc3a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fabb800e040 tx=0x7fabb8006ee0).stop 2024-01-21T19:48:10.143 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.148+0000 7fabce9ca700 1 -- 172.21.15.169:0/659932888 shutdown_connections 2024-01-21T19:48:10.143 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.148+0000 7fabce9ca700 1 --2- 172.21.15.169:0/659932888 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7faba802fd50 0x7faba8032210 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:10.143 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.148+0000 7fabce9ca700 1 --2- 172.21.15.169:0/659932888 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabc80703f0 0x7fabc80fc3a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:10.143 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.148+0000 7fabce9ca700 1 -- 172.21.15.169:0/659932888 >> 172.21.15.169:0/659932888 conn(0x7fabc80f63c0 msgr2=0x7fabc80f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:10.143 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.148+0000 7fabce9ca700 1 -- 172.21.15.169:0/659932888 shutdown_connections 2024-01-21T19:48:10.143 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.148+0000 7fabce9ca700 1 -- 172.21.15.169:0/659932888 wait complete. 2024-01-21T19:48:10.495 INFO:teuthology.orchestra.run.smithi169.stderr:Deploying crash service with default placement... 2024-01-21T19:48:10.495 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-01-21T19:48:10.877 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.881+0000 7fa9e427d700 1 Processor -- start 2024-01-21T19:48:10.877 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.881+0000 7fa9e427d700 1 -- start start 2024-01-21T19:48:10.877 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.882+0000 7fa9e427d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9dc0fab80 0x7fa9dc0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:10.877 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.882+0000 7fa9e427d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc0fb490 con 0x7fa9dc0fab80 2024-01-21T19:48:10.877 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.882+0000 7fa9e2019700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9dc0fab80 0x7fa9dc0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:10.877 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.882+0000 7fa9e2019700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9dc0fab80 0x7fa9dc0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49784/0 (socket says 172.21.15.169:49784) 2024-01-21T19:48:10.877 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.882+0000 7fa9e2019700 1 -- 172.21.15.169:0/116536790 learned_addr learned my addr 172.21.15.169:0/116536790 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:10.877 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.882+0000 7fa9e2019700 1 -- 172.21.15.169:0/116536790 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc0fb5d0 con 0x7fa9dc0fab80 2024-01-21T19:48:10.877 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.882+0000 7fa9e2019700 1 --2- 172.21.15.169:0/116536790 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9dc0fab80 0x7fa9dc0fafa0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fa9cc00b5f0 tx=0x7fa9cc006ee0).ready entity=mon.0 client_cookie=b24a1d0b764c0be5 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:10.877 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.882+0000 7fa9e1017700 1 -- 172.21.15.169:0/116536790 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9cc00d600 con 0x7fa9dc0fab80 2024-01-21T19:48:10.878 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.883+0000 7fa9e1017700 1 -- 172.21.15.169:0/116536790 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa9cc00dbd0 con 0x7fa9dc0fab80 2024-01-21T19:48:10.878 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.883+0000 7fa9e1017700 1 -- 172.21.15.169:0/116536790 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9cc011c00 con 0x7fa9dc0fab80 2024-01-21T19:48:10.878 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.883+0000 7fa9e427d700 1 -- 172.21.15.169:0/116536790 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9dc0fab80 msgr2=0x7fa9dc0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:10.878 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.883+0000 7fa9e427d700 1 --2- 172.21.15.169:0/116536790 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9dc0fab80 0x7fa9dc0fafa0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fa9cc00b5f0 tx=0x7fa9cc006ee0).stop 2024-01-21T19:48:10.878 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.883+0000 7fa9e427d700 1 -- 172.21.15.169:0/116536790 shutdown_connections 2024-01-21T19:48:10.878 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.883+0000 7fa9e427d700 1 --2- 172.21.15.169:0/116536790 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9dc0fab80 0x7fa9dc0fafa0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:10.878 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.883+0000 7fa9e427d700 1 -- 172.21.15.169:0/116536790 >> 172.21.15.169:0/116536790 conn(0x7fa9dc0f63c0 msgr2=0x7fa9dc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:10.878 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.884+0000 7fa9e427d700 1 -- 172.21.15.169:0/116536790 shutdown_connections 2024-01-21T19:48:10.878 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.884+0000 7fa9e427d700 1 -- 172.21.15.169:0/116536790 wait complete. 2024-01-21T19:48:10.880 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.884+0000 7fa9e427d700 1 Processor -- start 2024-01-21T19:48:10.880 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.884+0000 7fa9e427d700 1 -- start start 2024-01-21T19:48:10.880 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.884+0000 7fa9e427d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9dc0fab80 0x7fa9dc10ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:10.880 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.884+0000 7fa9e427d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc10d1b0 con 0x7fa9dc0fab80 2024-01-21T19:48:10.880 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.884+0000 7fa9e2019700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9dc0fab80 0x7fa9dc10ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:10.880 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.884+0000 7fa9e2019700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9dc0fab80 0x7fa9dc10ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49794/0 (socket says 172.21.15.169:49794) 2024-01-21T19:48:10.880 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.884+0000 7fa9e2019700 1 -- 172.21.15.169:0/2431842422 learned_addr learned my addr 172.21.15.169:0/2431842422 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:10.880 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.885+0000 7fa9e2019700 1 -- 172.21.15.169:0/2431842422 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9cc00b200 con 0x7fa9dc0fab80 2024-01-21T19:48:10.880 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.885+0000 7fa9e2019700 1 --2- 172.21.15.169:0/2431842422 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9dc0fab80 0x7fa9dc10ccc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa9cc00b1d0 tx=0x7fa9cc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:10.880 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.885+0000 7fa9caffd700 1 -- 172.21.15.169:0/2431842422 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9cc01b410 con 0x7fa9dc0fab80 2024-01-21T19:48:10.881 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.885+0000 7fa9caffd700 1 -- 172.21.15.169:0/2431842422 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa9cc01b9e0 con 0x7fa9dc0fab80 2024-01-21T19:48:10.881 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.885+0000 7fa9e427d700 1 -- 172.21.15.169:0/2431842422 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9dc10d3b0 con 0x7fa9dc0fab80 2024-01-21T19:48:10.881 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.885+0000 7fa9caffd700 1 -- 172.21.15.169:0/2431842422 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9cc01a840 con 0x7fa9dc0fab80 2024-01-21T19:48:10.881 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.885+0000 7fa9e427d700 1 -- 172.21.15.169:0/2431842422 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9dc10d7c0 con 0x7fa9dc0fab80 2024-01-21T19:48:10.881 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.886+0000 7fa9caffd700 1 -- 172.21.15.169:0/2431842422 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa9cc00f070 con 0x7fa9dc0fab80 2024-01-21T19:48:10.881 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.886+0000 7fa9e427d700 1 -- 172.21.15.169:0/2431842422 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9dc045ad0 con 0x7fa9dc0fab80 2024-01-21T19:48:10.881 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.886+0000 7fa9caffd700 1 --2- 172.21.15.169:0/2431842422 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fa9c402fd50 0x7fa9c4032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:10.881 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.886+0000 7fa9caffd700 1 -- 172.21.15.169:0/2431842422 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fa9cc041360 con 0x7fa9dc0fab80 2024-01-21T19:48:10.881 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.886+0000 7fa9e1818700 1 --2- 172.21.15.169:0/2431842422 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fa9c402fd50 0x7fa9c4032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:10.881 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.886+0000 7fa9e1818700 1 --2- 172.21.15.169:0/2431842422 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fa9c402fd50 0x7fa9c4032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fa9d80096a0 tx=0x7fa9d8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:10.883 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:10.890+0000 7fa9caffd700 1 -- 172.21.15.169:0/2431842422 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9cc022d20 con 0x7fa9dc0fab80 2024-01-21T19:48:11.069 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:11.076+0000 7fa9e427d700 1 -- 172.21.15.169:0/2431842422 --> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9dc10e020 con 0x7fa9c402fd50 2024-01-21T19:48:11.073 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:11.080+0000 7fa9caffd700 1 -- 172.21.15.169:0/2431842422 <== mgr.14114 v2:172.21.15.169:6800/2079278379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fa9dc10e020 con 0x7fa9c402fd50 2024-01-21T19:48:11.073 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: Scheduled crash update... 2024-01-21T19:48:11.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:11.081+0000 7fa9e427d700 1 -- 172.21.15.169:0/2431842422 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fa9c402fd50 msgr2=0x7fa9c4032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:11.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:11.081+0000 7fa9e427d700 1 --2- 172.21.15.169:0/2431842422 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fa9c402fd50 0x7fa9c4032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fa9d80096a0 tx=0x7fa9d8006b40).stop 2024-01-21T19:48:11.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:11.081+0000 7fa9e427d700 1 -- 172.21.15.169:0/2431842422 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9dc0fab80 msgr2=0x7fa9dc10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:11.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:11.081+0000 7fa9e427d700 1 --2- 172.21.15.169:0/2431842422 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9dc0fab80 0x7fa9dc10ccc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa9cc00b1d0 tx=0x7fa9cc006ee0).stop 2024-01-21T19:48:11.076 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:11.081+0000 7fa9e427d700 1 -- 172.21.15.169:0/2431842422 shutdown_connections 2024-01-21T19:48:11.076 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:11.081+0000 7fa9e427d700 1 --2- 172.21.15.169:0/2431842422 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7fa9c402fd50 0x7fa9c4032210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:11.076 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:11.081+0000 7fa9e427d700 1 --2- 172.21.15.169:0/2431842422 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa9dc0fab80 0x7fa9dc10ccc0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:11.076 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:11.081+0000 7fa9e427d700 1 -- 172.21.15.169:0/2431842422 >> 172.21.15.169:0/2431842422 conn(0x7fa9dc0f63c0 msgr2=0x7fa9dc070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:11.076 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:11.081+0000 7fa9e427d700 1 -- 172.21.15.169:0/2431842422 shutdown_connections 2024-01-21T19:48:11.076 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:11.081+0000 7fa9e427d700 1 -- 172.21.15.169:0/2431842422 wait complete. 2024-01-21T19:48:11.281 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:11 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:10.144573+0000 mgr.smithi169.espgue (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:48:11.281 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:11 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cephadm 2024-01-21T19:48:10.145097+0000 mgr.smithi169.espgue (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-01-21T19:48:11.281 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:11 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:10.147484+0000 mon.smithi169 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' 2024-01-21T19:48:11.281 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:11 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:11.080644+0000 mon.smithi169 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' 2024-01-21T19:48:11.680 INFO:teuthology.orchestra.run.smithi169.stderr:Enabling mgr prometheus module... 2024-01-21T19:48:11.680 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-01-21T19:48:12.070 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.074+0000 7f4706376700 1 Processor -- start 2024-01-21T19:48:12.070 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.075+0000 7f4706376700 1 -- start start 2024-01-21T19:48:12.070 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.075+0000 7f4706376700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47000703f0 0x7f47000fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:12.070 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.075+0000 7f4706376700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47000fee10 con 0x7f47000703f0 2024-01-21T19:48:12.070 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.075+0000 7f46fffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47000703f0 0x7f47000fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:12.070 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.075+0000 7f46fffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47000703f0 0x7f47000fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49800/0 (socket says 172.21.15.169:49800) 2024-01-21T19:48:12.070 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.075+0000 7f46fffff700 1 -- 172.21.15.169:0/754448793 learned_addr learned my addr 172.21.15.169:0/754448793 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:12.070 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.075+0000 7f46fffff700 1 -- 172.21.15.169:0/754448793 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47000fef50 con 0x7f47000703f0 2024-01-21T19:48:12.070 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.075+0000 7f46fffff700 1 --2- 172.21.15.169:0/754448793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47000703f0 0x7f47000fe920 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f46f000b5f0 tx=0x7f46f0006ee0).ready entity=mon.0 client_cookie=111c3591e1918568 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:12.070 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.076+0000 7f46feffd700 1 -- 172.21.15.169:0/754448793 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46f000d600 con 0x7f47000703f0 2024-01-21T19:48:12.070 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.076+0000 7f46feffd700 1 -- 172.21.15.169:0/754448793 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f46f000dc10 con 0x7f47000703f0 2024-01-21T19:48:12.070 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.076+0000 7f46feffd700 1 -- 172.21.15.169:0/754448793 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46f0011cd0 con 0x7f47000703f0 2024-01-21T19:48:12.071 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.076+0000 7f4706376700 1 -- 172.21.15.169:0/754448793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47000703f0 msgr2=0x7f47000fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:12.071 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.076+0000 7f4706376700 1 --2- 172.21.15.169:0/754448793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47000703f0 0x7f47000fe920 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f46f000b5f0 tx=0x7f46f0006ee0).stop 2024-01-21T19:48:12.074 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.077+0000 7f4706376700 1 -- 172.21.15.169:0/754448793 shutdown_connections 2024-01-21T19:48:12.074 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.077+0000 7f4706376700 1 --2- 172.21.15.169:0/754448793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47000703f0 0x7f47000fe920 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:12.074 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.077+0000 7f4706376700 1 -- 172.21.15.169:0/754448793 >> 172.21.15.169:0/754448793 conn(0x7f47000f63c0 msgr2=0x7f47000f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:12.074 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.077+0000 7f4706376700 1 -- 172.21.15.169:0/754448793 shutdown_connections 2024-01-21T19:48:12.074 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.077+0000 7f4706376700 1 -- 172.21.15.169:0/754448793 wait complete. 2024-01-21T19:48:12.074 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.077+0000 7f4706376700 1 Processor -- start 2024-01-21T19:48:12.074 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.078+0000 7f4706376700 1 -- start start 2024-01-21T19:48:12.074 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.078+0000 7f4706376700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47000703f0 0x7f470010aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:12.074 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.078+0000 7f4706376700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470010af70 con 0x7f47000703f0 2024-01-21T19:48:12.074 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.078+0000 7f46fffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47000703f0 0x7f470010aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:12.074 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.078+0000 7f46fffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47000703f0 0x7f470010aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49806/0 (socket says 172.21.15.169:49806) 2024-01-21T19:48:12.074 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.078+0000 7f46fffff700 1 -- 172.21.15.169:0/1044074029 learned_addr learned my addr 172.21.15.169:0/1044074029 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:12.074 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.078+0000 7f46fffff700 1 -- 172.21.15.169:0/1044074029 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f000b200 con 0x7f47000703f0 2024-01-21T19:48:12.074 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.078+0000 7f46fffff700 1 --2- 172.21.15.169:0/1044074029 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47000703f0 0x7f470010aa80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f46f0006a40 tx=0x7f46f0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:12.074 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.078+0000 7f46fcff9700 1 -- 172.21.15.169:0/1044074029 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46f0011c90 con 0x7f47000703f0 2024-01-21T19:48:12.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.078+0000 7f4706376700 1 -- 172.21.15.169:0/1044074029 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f470010b170 con 0x7f47000703f0 2024-01-21T19:48:12.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.078+0000 7f46fcff9700 1 -- 172.21.15.169:0/1044074029 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f46f001b8c0 con 0x7f47000703f0 2024-01-21T19:48:12.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.078+0000 7f46fcff9700 1 -- 172.21.15.169:0/1044074029 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46f001a8c0 con 0x7f47000703f0 2024-01-21T19:48:12.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.078+0000 7f4706376700 1 -- 172.21.15.169:0/1044074029 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f470010b580 con 0x7f47000703f0 2024-01-21T19:48:12.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.079+0000 7f46fcff9700 1 -- 172.21.15.169:0/1044074029 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f46f000f070 con 0x7f47000703f0 2024-01-21T19:48:12.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.079+0000 7f46fcff9700 1 --2- 172.21.15.169:0/1044074029 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f46ec02fd50 0x7f46ec032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:12.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.079+0000 7f46fcff9700 1 -- 172.21.15.169:0/1044074029 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f46f0041860 con 0x7f47000703f0 2024-01-21T19:48:12.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.080+0000 7f46ff7fe700 1 --2- 172.21.15.169:0/1044074029 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f46ec02fd50 0x7f46ec032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:12.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.080+0000 7f4706376700 1 -- 172.21.15.169:0/1044074029 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4700059100 con 0x7f47000703f0 2024-01-21T19:48:12.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.080+0000 7f46ff7fe700 1 --2- 172.21.15.169:0/1044074029 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f46ec02fd50 0x7f46ec032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f46f40096a0 tx=0x7f46f4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:12.077 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.084+0000 7f46fcff9700 1 -- 172.21.15.169:0/1044074029 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46f0016070 con 0x7f47000703f0 2024-01-21T19:48:12.272 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:12.278+0000 7f4706376700 1 -- 172.21.15.169:0/1044074029 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f47000ffc40 con 0x7f47000703f0 2024-01-21T19:48:13.065 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:13.071+0000 7f46fcff9700 1 -- 172.21.15.169:0/1044074029 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f46f00244b0 con 0x7f47000703f0 2024-01-21T19:48:13.065 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:13.072+0000 7f46fcff9700 1 -- 172.21.15.169:0/1044074029 <== mon.0 v2:172.21.15.169:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f46f0010720 con 0x7f47000703f0 2024-01-21T19:48:13.069 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:13.074+0000 7f4706376700 1 -- 172.21.15.169:0/1044074029 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f46ec02fd50 msgr2=0x7f46ec032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:13.069 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:13.074+0000 7f4706376700 1 --2- 172.21.15.169:0/1044074029 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f46ec02fd50 0x7f46ec032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f46f40096a0 tx=0x7f46f4006b40).stop 2024-01-21T19:48:13.069 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:13.074+0000 7f4706376700 1 -- 172.21.15.169:0/1044074029 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47000703f0 msgr2=0x7f470010aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:13.069 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:13.074+0000 7f4706376700 1 --2- 172.21.15.169:0/1044074029 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47000703f0 0x7f470010aa80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f46f0006a40 tx=0x7f46f0006ee0).stop 2024-01-21T19:48:13.069 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:13.074+0000 7f4706376700 1 -- 172.21.15.169:0/1044074029 shutdown_connections 2024-01-21T19:48:13.069 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:13.074+0000 7f4706376700 1 --2- 172.21.15.169:0/1044074029 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f46ec02fd50 0x7f46ec032210 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:13.069 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:13.074+0000 7f4706376700 1 --2- 172.21.15.169:0/1044074029 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f47000703f0 0x7f470010aa80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:13.069 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:13.074+0000 7f4706376700 1 -- 172.21.15.169:0/1044074029 >> 172.21.15.169:0/1044074029 conn(0x7f47000f63c0 msgr2=0x7f47000f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:13.069 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:13.075+0000 7f4706376700 1 -- 172.21.15.169:0/1044074029 shutdown_connections 2024-01-21T19:48:13.069 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:13.075+0000 7f4706376700 1 -- 172.21.15.169:0/1044074029 wait complete. 2024-01-21T19:48:13.096 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:12 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:11.077755+0000 mgr.smithi169.espgue (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:48:13.096 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:12 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cephadm 2024-01-21T19:48:11.078376+0000 mgr.smithi169.espgue (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-01-21T19:48:13.096 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:12 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:11.835660+0000 mon.smithi169 (mon.0) 59 : audit [INF] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' 2024-01-21T19:48:13.096 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:12 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:12.071046+0000 mon.smithi169 (mon.0) 60 : audit [INF] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' 2024-01-21T19:48:13.096 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:12 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:12.279971+0000 mon.smithi169 (mon.0) 61 : audit [INF] from='client.? 172.21.15.169:0/1044074029' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-21T19:48:13.096 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:12 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:12.715191+0000 mon.smithi169 (mon.0) 62 : audit [INF] from='mgr.14114 172.21.15.169:0/3231986151' entity='mgr.smithi169.espgue' 2024-01-21T19:48:13.698 INFO:teuthology.orchestra.run.smithi169.stderr:Deploying prometheus service with default placement... 2024-01-21T19:48:13.698 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-01-21T19:48:14.070 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.076+0000 7f1f7eaa3700 1 Processor -- start 2024-01-21T19:48:14.071 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.076+0000 7f1f7eaa3700 1 -- start start 2024-01-21T19:48:14.071 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.076+0000 7f1f7eaa3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f78070430 0x7f1f780fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:14.071 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.076+0000 7f1f7eaa3700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f780fedd0 con 0x7f1f78070430 2024-01-21T19:48:14.071 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.076+0000 7f1f7c83f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f78070430 0x7f1f780fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:14.071 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.076+0000 7f1f7c83f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f78070430 0x7f1f780fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49836/0 (socket says 172.21.15.169:49836) 2024-01-21T19:48:14.071 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.076+0000 7f1f7c83f700 1 -- 172.21.15.169:0/4229175926 learned_addr learned my addr 172.21.15.169:0/4229175926 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:14.071 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.077+0000 7f1f7c83f700 1 -- 172.21.15.169:0/4229175926 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f780fef10 con 0x7f1f78070430 2024-01-21T19:48:14.071 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.077+0000 7f1f7c83f700 1 --2- 172.21.15.169:0/4229175926 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f78070430 0x7f1f780fe8e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f1f6c00b5f0 tx=0x7f1f6c006ee0).ready entity=mon.0 client_cookie=2ede242338b50e7d server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:14.073 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.077+0000 7f1f777fe700 1 -- 172.21.15.169:0/4229175926 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f6c00d600 con 0x7f1f78070430 2024-01-21T19:48:14.073 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.077+0000 7f1f777fe700 1 -- 172.21.15.169:0/4229175926 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f6c00dc10 con 0x7f1f78070430 2024-01-21T19:48:14.073 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.077+0000 7f1f777fe700 1 -- 172.21.15.169:0/4229175926 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f6c011cd0 con 0x7f1f78070430 2024-01-21T19:48:14.073 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.078+0000 7f1f7eaa3700 1 -- 172.21.15.169:0/4229175926 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f78070430 msgr2=0x7f1f780fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:14.073 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.078+0000 7f1f7eaa3700 1 --2- 172.21.15.169:0/4229175926 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f78070430 0x7f1f780fe8e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f1f6c00b5f0 tx=0x7f1f6c006ee0).stop 2024-01-21T19:48:14.073 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.078+0000 7f1f7eaa3700 1 -- 172.21.15.169:0/4229175926 shutdown_connections 2024-01-21T19:48:14.073 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.078+0000 7f1f7eaa3700 1 --2- 172.21.15.169:0/4229175926 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f78070430 0x7f1f780fe8e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:14.073 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.078+0000 7f1f7eaa3700 1 -- 172.21.15.169:0/4229175926 >> 172.21.15.169:0/4229175926 conn(0x7f1f780f63c0 msgr2=0x7f1f780f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:14.073 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.078+0000 7f1f7eaa3700 1 -- 172.21.15.169:0/4229175926 shutdown_connections 2024-01-21T19:48:14.073 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.078+0000 7f1f7eaa3700 1 -- 172.21.15.169:0/4229175926 wait complete. 2024-01-21T19:48:14.073 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.078+0000 7f1f7eaa3700 1 Processor -- start 2024-01-21T19:48:14.073 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.078+0000 7f1f7eaa3700 1 -- start start 2024-01-21T19:48:14.074 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.079+0000 7f1f7eaa3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f78070430 0x7f1f7810ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:14.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.079+0000 7f1f7eaa3700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7810aff0 con 0x7f1f78070430 2024-01-21T19:48:14.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.079+0000 7f1f7c83f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f78070430 0x7f1f7810ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:14.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.079+0000 7f1f7c83f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f78070430 0x7f1f7810ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:49842/0 (socket says 172.21.15.169:49842) 2024-01-21T19:48:14.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.079+0000 7f1f7c83f700 1 -- 172.21.15.169:0/4118641796 learned_addr learned my addr 172.21.15.169:0/4118641796 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:14.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.079+0000 7f1f7c83f700 1 -- 172.21.15.169:0/4118641796 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6c00b200 con 0x7f1f78070430 2024-01-21T19:48:14.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.079+0000 7f1f7c83f700 1 --2- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f78070430 0x7f1f7810ab00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f1f6c006800 tx=0x7f1f6c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:14.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.079+0000 7f1f757fa700 1 -- 172.21.15.169:0/4118641796 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f6c011df0 con 0x7f1f78070430 2024-01-21T19:48:14.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.079+0000 7f1f7eaa3700 1 -- 172.21.15.169:0/4118641796 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f7810b1f0 con 0x7f1f78070430 2024-01-21T19:48:14.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.079+0000 7f1f757fa700 1 -- 172.21.15.169:0/4118641796 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f6c01b8c0 con 0x7f1f78070430 2024-01-21T19:48:14.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.079+0000 7f1f757fa700 1 -- 172.21.15.169:0/4118641796 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f6c01a830 con 0x7f1f78070430 2024-01-21T19:48:14.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.079+0000 7f1f7eaa3700 1 -- 172.21.15.169:0/4118641796 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f7810b600 con 0x7f1f78070430 2024-01-21T19:48:14.075 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.080+0000 7f1f757fa700 1 -- 172.21.15.169:0/4118641796 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f1f6c00f070 con 0x7f1f78070430 2024-01-21T19:48:14.076 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.080+0000 7f1f757fa700 1 --2- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f1f5c02fdb0 0x7f1f5c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:14.076 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.080+0000 7f1f757fa700 1 -- 172.21.15.169:0/4118641796 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f1f6c041e70 con 0x7f1f78070430 2024-01-21T19:48:14.076 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.080+0000 7f1f77fff700 1 -- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f1f5c02fdb0 msgr2=0x7f1f5c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/2079278379 2024-01-21T19:48:14.076 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.080+0000 7f1f77fff700 1 --2- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f1f5c02fdb0 0x7f1f5c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T19:48:14.076 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.080+0000 7f1f7eaa3700 1 -- 172.21.15.169:0/4118641796 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f78059100 con 0x7f1f78070430 2024-01-21T19:48:14.078 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.085+0000 7f1f757fa700 1 -- 172.21.15.169:0/4118641796 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f6c024410 con 0x7f1f78070430 2024-01-21T19:48:14.262 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.268+0000 7f1f7eaa3700 1 -- 172.21.15.169:0/4118641796 --> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f781048d0 con 0x7f1f5c02fdb0 2024-01-21T19:48:14.274 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.281+0000 7f1f77fff700 1 -- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f1f5c02fdb0 msgr2=0x7f1f5c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/2079278379 2024-01-21T19:48:14.274 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.281+0000 7f1f77fff700 1 --2- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f1f5c02fdb0 0x7f1f5c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T19:48:14.416 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:14 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:13.073561+0000 mon.smithi169 (mon.0) 63 : audit [INF] from='client.? 172.21.15.169:0/1044074029' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-21T19:48:14.416 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:14 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:13.073619+0000 mon.smithi169 (mon.0) 64 : cluster [DBG] mgrmap e9: smithi169.espgue(active, since 16s) 2024-01-21T19:48:14.675 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.682+0000 7f1f77fff700 1 -- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f1f5c02fdb0 msgr2=0x7f1f5c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/2079278379 2024-01-21T19:48:14.675 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:14.682+0000 7f1f77fff700 1 --2- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f1f5c02fdb0 0x7f1f5c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-21T19:48:15.476 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:15.483+0000 7f1f77fff700 1 -- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f1f5c02fdb0 msgr2=0x7f1f5c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/2079278379 2024-01-21T19:48:15.477 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:15.483+0000 7f1f77fff700 1 --2- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f1f5c02fdb0 0x7f1f5c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-21T19:48:17.078 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:17.084+0000 7f1f77fff700 1 -- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f1f5c02fdb0 msgr2=0x7f1f5c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/2079278379 2024-01-21T19:48:17.078 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:17.084+0000 7f1f77fff700 1 --2- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f1f5c02fdb0 0x7f1f5c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-21T19:48:18.624 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:18.629+0000 7f1f757fa700 1 -- 172.21.15.169:0/4118641796 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f1f6c010dd0 con 0x7f1f78070430 2024-01-21T19:48:18.625 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:18.629+0000 7f1f757fa700 1 -- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f1f5c02fdb0 msgr2=0x7f1f5c032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T19:48:18.625 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:18.629+0000 7f1f757fa700 1 --2- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f1f5c02fdb0 0x7f1f5c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:48:18.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:18 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:18.089850+0000 mon.smithi169 (mon.0) 65 : cluster [INF] Active manager daemon smithi169.espgue restarted 2024-01-21T19:48:18.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:18 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:18.090355+0000 mon.smithi169 (mon.0) 66 : cluster [INF] Activating manager daemon smithi169.espgue 2024-01-21T19:48:18.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:18 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:18.092045+0000 mon.smithi169 (mon.0) 67 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-01-21T19:48:19.629 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:19.633+0000 7f1f757fa700 1 -- 172.21.15.169:0/4118641796 <== mon.0 v2:172.21.15.169:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f1f6c044aa0 con 0x7f1f78070430 2024-01-21T19:48:19.629 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:19.633+0000 7f1f757fa700 1 --2- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f1f5c033c20 0x7f1f5c036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:19.629 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:19.633+0000 7f1f757fa700 1 -- 172.21.15.169:0/4118641796 --> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f781048d0 con 0x7f1f5c033c20 2024-01-21T19:48:19.629 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:19.634+0000 7f1f77fff700 1 --2- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f1f5c033c20 0x7f1f5c036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:19.629 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:19.634+0000 7f1f77fff700 1 --2- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f1f5c033c20 0x7f1f5c036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f1f640036d0 tx=0x7f1f64006cb0).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:19.634 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:19.640+0000 7f1f757fa700 1 -- 172.21.15.169:0/4118641796 <== mgr.14142 v2:172.21.15.169:6800/528052065 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f1f781048d0 con 0x7f1f5c033c20 2024-01-21T19:48:19.635 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-01-21T19:48:19.637 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:19.642+0000 7f1f7eaa3700 1 -- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f1f5c033c20 msgr2=0x7f1f5c036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:19.637 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:19.642+0000 7f1f7eaa3700 1 --2- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f1f5c033c20 0x7f1f5c036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f1f640036d0 tx=0x7f1f64006cb0).stop 2024-01-21T19:48:19.637 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:19.642+0000 7f1f7eaa3700 1 -- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f78070430 msgr2=0x7f1f7810ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:19.637 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:19.642+0000 7f1f7eaa3700 1 --2- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f78070430 0x7f1f7810ab00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f1f6c006800 tx=0x7f1f6c006ee0).stop 2024-01-21T19:48:19.637 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:19.643+0000 7f1f7eaa3700 1 -- 172.21.15.169:0/4118641796 shutdown_connections 2024-01-21T19:48:19.637 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:19.643+0000 7f1f7eaa3700 1 --2- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f1f5c033c20 0x7f1f5c036010 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:19.638 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:19.643+0000 7f1f7eaa3700 1 --2- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f78070430 0x7f1f7810ab00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:19.638 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:19.643+0000 7f1f7eaa3700 1 --2- 172.21.15.169:0/4118641796 >> [v2:172.21.15.169:6800/2079278379,v1:172.21.15.169:6801/2079278379] conn(0x7f1f5c02fdb0 0x7f1f5c032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:48:19.638 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:19.643+0000 7f1f7eaa3700 1 -- 172.21.15.169:0/4118641796 >> 172.21.15.169:0/4118641796 conn(0x7f1f780f63c0 msgr2=0x7f1f780f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:19.638 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:19.643+0000 7f1f7eaa3700 1 -- 172.21.15.169:0/4118641796 shutdown_connections 2024-01-21T19:48:19.638 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:19.643+0000 7f1f7eaa3700 1 -- 172.21.15.169:0/4118641796 wait complete. 2024-01-21T19:48:19.810 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:19 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:18.631993+0000 mon.smithi169 (mon.0) 68 : cluster [DBG] mgrmap e10: smithi169.espgue(active, starting, since 0.541733s) 2024-01-21T19:48:19.810 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:19 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:18.633345+0000 mon.smithi169 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.169:0/778638721' entity='mgr.smithi169.espgue' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-01-21T19:48:19.810 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:19 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:18.633921+0000 mon.smithi169 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.169:0/778638721' entity='mgr.smithi169.espgue' cmd=[{"prefix": "mgr metadata", "who": "smithi169.espgue", "id": "smithi169.espgue"}]: dispatch 2024-01-21T19:48:19.810 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:19 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:18.634624+0000 mon.smithi169 (mon.0) 71 : audit [DBG] from='mgr.14142 172.21.15.169:0/778638721' entity='mgr.smithi169.espgue' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T19:48:19.810 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:19 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:18.634799+0000 mon.smithi169 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.169:0/778638721' entity='mgr.smithi169.espgue' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T19:48:19.810 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:19 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:18.634968+0000 mon.smithi169 (mon.0) 73 : audit [DBG] from='mgr.14142 172.21.15.169:0/778638721' entity='mgr.smithi169.espgue' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T19:48:19.810 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:19 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:18.636984+0000 mon.smithi169 (mon.0) 74 : cluster [INF] Manager daemon smithi169.espgue is now available 2024-01-21T19:48:19.810 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:19 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:18.736978+0000 mon.smithi169 (mon.0) 75 : audit [DBG] from='mgr.14142 172.21.15.169:0/778638721' entity='mgr.smithi169.espgue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:48:19.810 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:19 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:18.738492+0000 mon.smithi169 (mon.0) 76 : audit [DBG] from='mgr.14142 172.21.15.169:0/778638721' entity='mgr.smithi169.espgue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:48:19.810 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:19 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:18.752683+0000 mon.smithi169 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.169:0/778638721' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/mirror_snapshot_schedule"}]: dispatch 2024-01-21T19:48:19.810 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:19 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:18.756346+0000 mon.smithi169 (mon.0) 78 : audit [INF] from='mgr.14142 172.21.15.169:0/778638721' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/trash_purge_schedule"}]: dispatch 2024-01-21T19:48:20.486 INFO:teuthology.orchestra.run.smithi169.stderr:Deploying grafana service with default placement... 2024-01-21T19:48:20.486 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-01-21T19:48:20.720 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:20 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:19.637666+0000 mon.smithi169 (mon.0) 79 : cluster [DBG] mgrmap e11: smithi169.espgue(active, since 1.54741s) 2024-01-21T19:48:20.720 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:20 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cephadm 2024-01-21T19:48:19.638336+0000 mgr.smithi169.espgue (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-01-21T19:48:20.720 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:20 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:19.642002+0000 mon.smithi169 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.169:0/778638721' entity='mgr.smithi169.espgue' 2024-01-21T19:48:20.720 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:20 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:19.673292+0000 mon.smithi169 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.169:0/778638721' entity='mgr.smithi169.espgue' 2024-01-21T19:48:21.029 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.035+0000 7fcb9759e700 1 Processor -- start 2024-01-21T19:48:21.030 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.036+0000 7fcb9759e700 1 -- start start 2024-01-21T19:48:21.030 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.036+0000 7fcb9759e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb980fa9d0 0x7fcb980fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:21.030 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.036+0000 7fcb9759e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb980fb2e0 con 0x7fcb980fa9d0 2024-01-21T19:48:21.030 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.036+0000 7fcb9659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb980fa9d0 0x7fcb980fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:21.032 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.036+0000 7fcb9659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb980fa9d0 0x7fcb980fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47194/0 (socket says 172.21.15.169:47194) 2024-01-21T19:48:21.032 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.036+0000 7fcb9659c700 1 -- 172.21.15.169:0/945468834 learned_addr learned my addr 172.21.15.169:0/945468834 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:21.032 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.037+0000 7fcb9659c700 1 -- 172.21.15.169:0/945468834 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb980fb420 con 0x7fcb980fa9d0 2024-01-21T19:48:21.032 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.037+0000 7fcb9659c700 1 --2- 172.21.15.169:0/945468834 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb980fa9d0 0x7fcb980fadf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fcb8c00b5f0 tx=0x7fcb8c006ee0).ready entity=mon.0 client_cookie=8c228cfe302516b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:21.032 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.037+0000 7fcb9559a700 1 -- 172.21.15.169:0/945468834 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb8c00d600 con 0x7fcb980fa9d0 2024-01-21T19:48:21.032 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.037+0000 7fcb9559a700 1 -- 172.21.15.169:0/945468834 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb8c00dc10 con 0x7fcb980fa9d0 2024-01-21T19:48:21.033 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.037+0000 7fcb9559a700 1 -- 172.21.15.169:0/945468834 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb8c011cd0 con 0x7fcb980fa9d0 2024-01-21T19:48:21.033 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.037+0000 7fcb9759e700 1 -- 172.21.15.169:0/945468834 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb980fa9d0 msgr2=0x7fcb980fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:21.033 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.037+0000 7fcb9759e700 1 --2- 172.21.15.169:0/945468834 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb980fa9d0 0x7fcb980fadf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fcb8c00b5f0 tx=0x7fcb8c006ee0).stop 2024-01-21T19:48:21.033 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.038+0000 7fcb9759e700 1 -- 172.21.15.169:0/945468834 shutdown_connections 2024-01-21T19:48:21.033 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.038+0000 7fcb9759e700 1 --2- 172.21.15.169:0/945468834 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb980fa9d0 0x7fcb980fadf0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:21.033 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.038+0000 7fcb9759e700 1 -- 172.21.15.169:0/945468834 >> 172.21.15.169:0/945468834 conn(0x7fcb980f6210 msgr2=0x7fcb980f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:21.033 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.038+0000 7fcb9759e700 1 -- 172.21.15.169:0/945468834 shutdown_connections 2024-01-21T19:48:21.033 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.038+0000 7fcb9759e700 1 -- 172.21.15.169:0/945468834 wait complete. 2024-01-21T19:48:21.033 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.038+0000 7fcb9759e700 1 Processor -- start 2024-01-21T19:48:21.033 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.038+0000 7fcb9759e700 1 -- start start 2024-01-21T19:48:21.033 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.039+0000 7fcb9759e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb980fa9d0 0x7fcb9810f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:21.033 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.039+0000 7fcb9759e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9810b390 con 0x7fcb980fa9d0 2024-01-21T19:48:21.033 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.039+0000 7fcb9659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb980fa9d0 0x7fcb9810f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:21.033 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.039+0000 7fcb9659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb980fa9d0 0x7fcb9810f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47202/0 (socket says 172.21.15.169:47202) 2024-01-21T19:48:21.033 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.039+0000 7fcb9659c700 1 -- 172.21.15.169:0/2495173667 learned_addr learned my addr 172.21.15.169:0/2495173667 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:21.034 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.039+0000 7fcb9659c700 1 -- 172.21.15.169:0/2495173667 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8c00b200 con 0x7fcb980fa9d0 2024-01-21T19:48:21.035 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.039+0000 7fcb9659c700 1 --2- 172.21.15.169:0/2495173667 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb980fa9d0 0x7fcb9810f9a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fcb8c00dfb0 tx=0x7fcb8c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:21.035 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.039+0000 7fcb86ffd700 1 -- 172.21.15.169:0/2495173667 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb8c011df0 con 0x7fcb980fa9d0 2024-01-21T19:48:21.035 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.039+0000 7fcb9759e700 1 -- 172.21.15.169:0/2495173667 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb980704b0 con 0x7fcb980fa9d0 2024-01-21T19:48:21.035 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.039+0000 7fcb86ffd700 1 -- 172.21.15.169:0/2495173667 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb8c01b8c0 con 0x7fcb980fa9d0 2024-01-21T19:48:21.035 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.039+0000 7fcb86ffd700 1 -- 172.21.15.169:0/2495173667 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb8c01a8c0 con 0x7fcb980fa9d0 2024-01-21T19:48:21.035 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.039+0000 7fcb9759e700 1 -- 172.21.15.169:0/2495173667 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb9810fe90 con 0x7fcb980fa9d0 2024-01-21T19:48:21.035 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.040+0000 7fcb86ffd700 1 -- 172.21.15.169:0/2495173667 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fcb8c00f070 con 0x7fcb980fa9d0 2024-01-21T19:48:21.035 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.040+0000 7fcb86ffd700 1 --2- 172.21.15.169:0/2495173667 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fcb7c02fe60 0x7fcb7c032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:21.035 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.040+0000 7fcb86ffd700 1 -- 172.21.15.169:0/2495173667 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fcb8c0410d0 con 0x7fcb980fa9d0 2024-01-21T19:48:21.035 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.040+0000 7fcb9759e700 1 -- 172.21.15.169:0/2495173667 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb98045ad0 con 0x7fcb980fa9d0 2024-01-21T19:48:21.035 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.040+0000 7fcb95d9b700 1 --2- 172.21.15.169:0/2495173667 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fcb7c02fe60 0x7fcb7c032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:21.035 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.041+0000 7fcb95d9b700 1 --2- 172.21.15.169:0/2495173667 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fcb7c02fe60 0x7fcb7c032320 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fcb880096a0 tx=0x7fcb88006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:21.037 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.044+0000 7fcb86ffd700 1 -- 172.21.15.169:0/2495173667 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb8c016070 con 0x7fcb980fa9d0 2024-01-21T19:48:21.215 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.222+0000 7fcb9759e700 1 -- 172.21.15.169:0/2495173667 --> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb98057f10 con 0x7fcb7c02fe60 2024-01-21T19:48:21.219 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.226+0000 7fcb86ffd700 1 -- 172.21.15.169:0/2495173667 <== mgr.14142 v2:172.21.15.169:6800/528052065 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fcb98057f10 con 0x7fcb7c02fe60 2024-01-21T19:48:21.219 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-01-21T19:48:21.221 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.227+0000 7fcb9759e700 1 -- 172.21.15.169:0/2495173667 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fcb7c02fe60 msgr2=0x7fcb7c032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:21.221 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.227+0000 7fcb9759e700 1 --2- 172.21.15.169:0/2495173667 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fcb7c02fe60 0x7fcb7c032320 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fcb880096a0 tx=0x7fcb88006b40).stop 2024-01-21T19:48:21.222 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.227+0000 7fcb9759e700 1 -- 172.21.15.169:0/2495173667 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb980fa9d0 msgr2=0x7fcb9810f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:21.222 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.227+0000 7fcb9759e700 1 --2- 172.21.15.169:0/2495173667 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb980fa9d0 0x7fcb9810f9a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fcb8c00dfb0 tx=0x7fcb8c006ee0).stop 2024-01-21T19:48:21.222 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.227+0000 7fcb9759e700 1 -- 172.21.15.169:0/2495173667 shutdown_connections 2024-01-21T19:48:21.222 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.227+0000 7fcb9759e700 1 --2- 172.21.15.169:0/2495173667 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fcb7c02fe60 0x7fcb7c032320 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:21.222 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.227+0000 7fcb9759e700 1 --2- 172.21.15.169:0/2495173667 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb980fa9d0 0x7fcb9810f9a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:21.222 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.227+0000 7fcb9759e700 1 -- 172.21.15.169:0/2495173667 >> 172.21.15.169:0/2495173667 conn(0x7fcb980f6210 msgr2=0x7fcb980f7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:21.222 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.228+0000 7fcb9759e700 1 -- 172.21.15.169:0/2495173667 shutdown_connections 2024-01-21T19:48:21.222 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.228+0000 7fcb9759e700 1 -- 172.21.15.169:0/2495173667 wait complete. 2024-01-21T19:48:21.577 INFO:teuthology.orchestra.run.smithi169.stderr:Deploying node-exporter service with default placement... 2024-01-21T19:48:21.577 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-01-21T19:48:21.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:21 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:20.674256+0000 mon.smithi169 (mon.0) 82 : cluster [DBG] mgrmap e12: smithi169.espgue(active, since 2s) 2024-01-21T19:48:21.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:21 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:21.226739+0000 mon.smithi169 (mon.0) 83 : audit [INF] from='mgr.14142 172.21.15.169:0/778638721' entity='mgr.smithi169.espgue' 2024-01-21T19:48:21.941 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.946+0000 7fa088642700 1 Processor -- start 2024-01-21T19:48:21.941 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.946+0000 7fa088642700 1 -- start start 2024-01-21T19:48:21.941 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.947+0000 7fa088642700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0800fab80 0x7fa0800fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:21.941 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.947+0000 7fa088642700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0800fb490 con 0x7fa0800fab80 2024-01-21T19:48:21.941 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.947+0000 7fa0863de700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0800fab80 0x7fa0800fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:21.941 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.947+0000 7fa0863de700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0800fab80 0x7fa0800fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47218/0 (socket says 172.21.15.169:47218) 2024-01-21T19:48:21.942 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.947+0000 7fa0863de700 1 -- 172.21.15.169:0/2849118337 learned_addr learned my addr 172.21.15.169:0/2849118337 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:21.942 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.947+0000 7fa0863de700 1 -- 172.21.15.169:0/2849118337 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0800fb5d0 con 0x7fa0800fab80 2024-01-21T19:48:21.942 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.947+0000 7fa0863de700 1 --2- 172.21.15.169:0/2849118337 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0800fab80 0x7fa0800fafa0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fa07000b5f0 tx=0x7fa070006ee0).ready entity=mon.0 client_cookie=8fadb0aae99ef15a server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:21.942 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.947+0000 7fa0853dc700 1 -- 172.21.15.169:0/2849118337 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa07000d600 con 0x7fa0800fab80 2024-01-21T19:48:21.942 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.947+0000 7fa0853dc700 1 -- 172.21.15.169:0/2849118337 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa07000dc10 con 0x7fa0800fab80 2024-01-21T19:48:21.942 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.947+0000 7fa0853dc700 1 -- 172.21.15.169:0/2849118337 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa070011cd0 con 0x7fa0800fab80 2024-01-21T19:48:21.942 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.948+0000 7fa088642700 1 -- 172.21.15.169:0/2849118337 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0800fab80 msgr2=0x7fa0800fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:21.942 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.948+0000 7fa088642700 1 --2- 172.21.15.169:0/2849118337 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0800fab80 0x7fa0800fafa0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fa07000b5f0 tx=0x7fa070006ee0).stop 2024-01-21T19:48:21.945 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.948+0000 7fa088642700 1 -- 172.21.15.169:0/2849118337 shutdown_connections 2024-01-21T19:48:21.945 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.948+0000 7fa088642700 1 --2- 172.21.15.169:0/2849118337 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0800fab80 0x7fa0800fafa0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:21.945 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.948+0000 7fa088642700 1 -- 172.21.15.169:0/2849118337 >> 172.21.15.169:0/2849118337 conn(0x7fa0800f63c0 msgr2=0x7fa0800f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:21.945 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.948+0000 7fa088642700 1 -- 172.21.15.169:0/2849118337 shutdown_connections 2024-01-21T19:48:21.945 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.948+0000 7fa088642700 1 -- 172.21.15.169:0/2849118337 wait complete. 2024-01-21T19:48:21.945 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.949+0000 7fa088642700 1 Processor -- start 2024-01-21T19:48:21.945 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.949+0000 7fa088642700 1 -- start start 2024-01-21T19:48:21.945 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.949+0000 7fa088642700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0800fab80 0x7fa080109930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:21.945 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.949+0000 7fa088642700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa080109e20 con 0x7fa0800fab80 2024-01-21T19:48:21.945 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.949+0000 7fa0863de700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0800fab80 0x7fa080109930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:21.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.949+0000 7fa0863de700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0800fab80 0x7fa080109930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47232/0 (socket says 172.21.15.169:47232) 2024-01-21T19:48:21.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.950+0000 7fa0863de700 1 -- 172.21.15.169:0/4200090791 learned_addr learned my addr 172.21.15.169:0/4200090791 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:21.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.950+0000 7fa0863de700 1 -- 172.21.15.169:0/4200090791 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07000b200 con 0x7fa0800fab80 2024-01-21T19:48:21.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.950+0000 7fa0863de700 1 --2- 172.21.15.169:0/4200090791 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0800fab80 0x7fa080109930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fa07000e040 tx=0x7fa070006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:21.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.950+0000 7fa06effd700 1 -- 172.21.15.169:0/4200090791 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa07001b410 con 0x7fa0800fab80 2024-01-21T19:48:21.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.950+0000 7fa06effd700 1 -- 172.21.15.169:0/4200090791 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa07001ba20 con 0x7fa0800fab80 2024-01-21T19:48:21.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.950+0000 7fa088642700 1 -- 172.21.15.169:0/4200090791 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa08010a020 con 0x7fa0800fab80 2024-01-21T19:48:21.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.950+0000 7fa06effd700 1 -- 172.21.15.169:0/4200090791 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa07001a910 con 0x7fa0800fab80 2024-01-21T19:48:21.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.950+0000 7fa088642700 1 -- 172.21.15.169:0/4200090791 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa08010a430 con 0x7fa0800fab80 2024-01-21T19:48:21.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.951+0000 7fa06effd700 1 -- 172.21.15.169:0/4200090791 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fa07000f070 con 0x7fa0800fab80 2024-01-21T19:48:21.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.951+0000 7fa06effd700 1 --2- 172.21.15.169:0/4200090791 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fa06802feb0 0x7fa068032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:21.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.951+0000 7fa06effd700 1 -- 172.21.15.169:0/4200090791 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fa0700411f0 con 0x7fa0800fab80 2024-01-21T19:48:21.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.951+0000 7fa088642700 1 -- 172.21.15.169:0/4200090791 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa080045ad0 con 0x7fa0800fab80 2024-01-21T19:48:21.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.951+0000 7fa085bdd700 1 --2- 172.21.15.169:0/4200090791 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fa06802feb0 0x7fa068032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:21.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.952+0000 7fa085bdd700 1 --2- 172.21.15.169:0/4200090791 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fa06802feb0 0x7fa068032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fa07c0096a0 tx=0x7fa07c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:21.948 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:21.955+0000 7fa06effd700 1 -- 172.21.15.169:0/4200090791 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa070016080 con 0x7fa0800fab80 2024-01-21T19:48:22.125 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.132+0000 7fa088642700 1 -- 172.21.15.169:0/4200090791 --> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fa08010ad10 con 0x7fa06802feb0 2024-01-21T19:48:22.129 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.135+0000 7fa06effd700 1 -- 172.21.15.169:0/4200090791 <== mgr.14142 v2:172.21.15.169:6800/528052065 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fa08010ad10 con 0x7fa06802feb0 2024-01-21T19:48:22.129 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-01-21T19:48:22.131 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.137+0000 7fa088642700 1 -- 172.21.15.169:0/4200090791 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fa06802feb0 msgr2=0x7fa068032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:22.131 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.137+0000 7fa088642700 1 --2- 172.21.15.169:0/4200090791 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fa06802feb0 0x7fa068032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fa07c0096a0 tx=0x7fa07c006b40).stop 2024-01-21T19:48:22.132 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.137+0000 7fa088642700 1 -- 172.21.15.169:0/4200090791 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0800fab80 msgr2=0x7fa080109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:22.132 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.137+0000 7fa088642700 1 --2- 172.21.15.169:0/4200090791 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0800fab80 0x7fa080109930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fa07000e040 tx=0x7fa070006ee0).stop 2024-01-21T19:48:22.132 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.137+0000 7fa088642700 1 -- 172.21.15.169:0/4200090791 shutdown_connections 2024-01-21T19:48:22.132 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.137+0000 7fa088642700 1 --2- 172.21.15.169:0/4200090791 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fa06802feb0 0x7fa068032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:22.132 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.137+0000 7fa088642700 1 --2- 172.21.15.169:0/4200090791 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0800fab80 0x7fa080109930 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:22.132 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.137+0000 7fa088642700 1 -- 172.21.15.169:0/4200090791 >> 172.21.15.169:0/4200090791 conn(0x7fa0800f63c0 msgr2=0x7fa0800f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:22.132 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.138+0000 7fa088642700 1 -- 172.21.15.169:0/4200090791 shutdown_connections 2024-01-21T19:48:22.132 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.138+0000 7fa088642700 1 -- 172.21.15.169:0/4200090791 wait complete. 2024-01-21T19:48:22.485 INFO:teuthology.orchestra.run.smithi169.stderr:Deploying alertmanager service with default placement... 2024-01-21T19:48:22.485 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-01-21T19:48:22.872 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.875+0000 7eff489b0700 1 Processor -- start 2024-01-21T19:48:22.872 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.875+0000 7eff489b0700 1 -- start start 2024-01-21T19:48:22.872 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.875+0000 7eff489b0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff440fab80 0x7eff440fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:22.872 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.875+0000 7eff489b0700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff440fb490 con 0x7eff440fab80 2024-01-21T19:48:22.872 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.875+0000 7eff4259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff440fab80 0x7eff440fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:22.872 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.875+0000 7eff4259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff440fab80 0x7eff440fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47248/0 (socket says 172.21.15.169:47248) 2024-01-21T19:48:22.872 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.875+0000 7eff4259c700 1 -- 172.21.15.169:0/242002521 learned_addr learned my addr 172.21.15.169:0/242002521 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:22.872 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.876+0000 7eff4259c700 1 -- 172.21.15.169:0/242002521 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff440fb5d0 con 0x7eff440fab80 2024-01-21T19:48:22.873 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.876+0000 7eff4259c700 1 --2- 172.21.15.169:0/242002521 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff440fab80 0x7eff440fafa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7eff3800b5f0 tx=0x7eff38006ee0).ready entity=mon.0 client_cookie=af208543439ad0d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:22.873 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.876+0000 7eff4159a700 1 -- 172.21.15.169:0/242002521 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff3800d600 con 0x7eff440fab80 2024-01-21T19:48:22.873 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.876+0000 7eff4159a700 1 -- 172.21.15.169:0/242002521 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff3800dc10 con 0x7eff440fab80 2024-01-21T19:48:22.873 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.876+0000 7eff4159a700 1 -- 172.21.15.169:0/242002521 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff38011cd0 con 0x7eff440fab80 2024-01-21T19:48:22.873 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.877+0000 7eff489b0700 1 -- 172.21.15.169:0/242002521 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff440fab80 msgr2=0x7eff440fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:22.873 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.877+0000 7eff489b0700 1 --2- 172.21.15.169:0/242002521 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff440fab80 0x7eff440fafa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7eff3800b5f0 tx=0x7eff38006ee0).stop 2024-01-21T19:48:22.873 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.877+0000 7eff489b0700 1 -- 172.21.15.169:0/242002521 shutdown_connections 2024-01-21T19:48:22.873 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.877+0000 7eff489b0700 1 --2- 172.21.15.169:0/242002521 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff440fab80 0x7eff440fafa0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:22.873 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.877+0000 7eff489b0700 1 -- 172.21.15.169:0/242002521 >> 172.21.15.169:0/242002521 conn(0x7eff440f63c0 msgr2=0x7eff440f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:22.873 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.877+0000 7eff489b0700 1 -- 172.21.15.169:0/242002521 shutdown_connections 2024-01-21T19:48:22.873 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.877+0000 7eff489b0700 1 -- 172.21.15.169:0/242002521 wait complete. 2024-01-21T19:48:22.873 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.877+0000 7eff489b0700 1 Processor -- start 2024-01-21T19:48:22.873 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.877+0000 7eff489b0700 1 -- start start 2024-01-21T19:48:22.873 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.877+0000 7eff489b0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff440fab80 0x7eff4410cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:22.873 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.877+0000 7eff489b0700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4410d230 con 0x7eff440fab80 2024-01-21T19:48:22.873 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.877+0000 7eff4259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff440fab80 0x7eff4410cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:22.874 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.878+0000 7eff4259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff440fab80 0x7eff4410cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47250/0 (socket says 172.21.15.169:47250) 2024-01-21T19:48:22.874 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.878+0000 7eff4259c700 1 -- 172.21.15.169:0/3356469783 learned_addr learned my addr 172.21.15.169:0/3356469783 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:22.874 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.878+0000 7eff4259c700 1 -- 172.21.15.169:0/3356469783 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3800b200 con 0x7eff440fab80 2024-01-21T19:48:22.874 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.878+0000 7eff4259c700 1 --2- 172.21.15.169:0/3356469783 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff440fab80 0x7eff4410cd40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7eff3800dfb0 tx=0x7eff38006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:22.874 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.878+0000 7eff32ffd700 1 -- 172.21.15.169:0/3356469783 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff38011df0 con 0x7eff440fab80 2024-01-21T19:48:22.874 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.878+0000 7eff489b0700 1 -- 172.21.15.169:0/3356469783 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff4410d430 con 0x7eff440fab80 2024-01-21T19:48:22.874 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.878+0000 7eff32ffd700 1 -- 172.21.15.169:0/3356469783 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff3801b8c0 con 0x7eff440fab80 2024-01-21T19:48:22.874 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.878+0000 7eff32ffd700 1 -- 172.21.15.169:0/3356469783 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff3801a8c0 con 0x7eff440fab80 2024-01-21T19:48:22.874 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.878+0000 7eff489b0700 1 -- 172.21.15.169:0/3356469783 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff4410d840 con 0x7eff440fab80 2024-01-21T19:48:22.874 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.879+0000 7eff32ffd700 1 -- 172.21.15.169:0/3356469783 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7eff3800f070 con 0x7eff440fab80 2024-01-21T19:48:22.875 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.879+0000 7eff32ffd700 1 --2- 172.21.15.169:0/3356469783 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7eff2802fe60 0x7eff28032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:22.875 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.879+0000 7eff32ffd700 1 -- 172.21.15.169:0/3356469783 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7eff38041180 con 0x7eff440fab80 2024-01-21T19:48:22.875 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.879+0000 7eff489b0700 1 -- 172.21.15.169:0/3356469783 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff44045ad0 con 0x7eff440fab80 2024-01-21T19:48:22.875 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.879+0000 7eff41d9b700 1 --2- 172.21.15.169:0/3356469783 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7eff2802fe60 0x7eff28032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:22.875 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.880+0000 7eff41d9b700 1 --2- 172.21.15.169:0/3356469783 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7eff2802fe60 0x7eff28032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7eff340096a0 tx=0x7eff34006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:22.877 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:22.884+0000 7eff32ffd700 1 -- 172.21.15.169:0/3356469783 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7eff38016070 con 0x7eff440fab80 2024-01-21T19:48:23.054 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.061+0000 7eff489b0700 1 -- 172.21.15.169:0/3356469783 --> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7eff4410ed70 con 0x7eff2802fe60 2024-01-21T19:48:23.061 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.067+0000 7eff32ffd700 1 -- 172.21.15.169:0/3356469783 <== mgr.14142 v2:172.21.15.169:6800/528052065 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7eff4410ed70 con 0x7eff2802fe60 2024-01-21T19:48:23.061 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-01-21T19:48:23.063 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.069+0000 7eff489b0700 1 -- 172.21.15.169:0/3356469783 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7eff2802fe60 msgr2=0x7eff28032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:23.063 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.069+0000 7eff489b0700 1 --2- 172.21.15.169:0/3356469783 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7eff2802fe60 0x7eff28032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7eff340096a0 tx=0x7eff34006b40).stop 2024-01-21T19:48:23.063 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.069+0000 7eff489b0700 1 -- 172.21.15.169:0/3356469783 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff440fab80 msgr2=0x7eff4410cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:23.063 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.069+0000 7eff489b0700 1 --2- 172.21.15.169:0/3356469783 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff440fab80 0x7eff4410cd40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7eff3800dfb0 tx=0x7eff38006ee0).stop 2024-01-21T19:48:23.063 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.069+0000 7eff489b0700 1 -- 172.21.15.169:0/3356469783 shutdown_connections 2024-01-21T19:48:23.064 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.069+0000 7eff489b0700 1 --2- 172.21.15.169:0/3356469783 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7eff2802fe60 0x7eff28032320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:23.064 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.069+0000 7eff489b0700 1 --2- 172.21.15.169:0/3356469783 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7eff440fab80 0x7eff4410cd40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:23.064 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.069+0000 7eff489b0700 1 -- 172.21.15.169:0/3356469783 >> 172.21.15.169:0/3356469783 conn(0x7eff440f63c0 msgr2=0x7eff44070010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:23.064 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.069+0000 7eff489b0700 1 -- 172.21.15.169:0/3356469783 shutdown_connections 2024-01-21T19:48:23.064 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.069+0000 7eff489b0700 1 -- 172.21.15.169:0/3356469783 wait complete. 2024-01-21T19:48:23.081 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:22 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:21.224034+0000 mgr.smithi169.espgue (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:48:23.081 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:22 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cephadm 2024-01-21T19:48:21.224525+0000 mgr.smithi169.espgue (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-01-21T19:48:23.081 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:22 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:22.136589+0000 mon.smithi169 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.169:0/778638721' entity='mgr.smithi169.espgue' 2024-01-21T19:48:23.434 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-01-21T19:48:23.791 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.795+0000 7f079f59e700 1 Processor -- start 2024-01-21T19:48:23.791 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.795+0000 7f079f59e700 1 -- start start 2024-01-21T19:48:23.791 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.796+0000 7f079f59e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f07a0070430 0x7f07a00fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:23.792 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.796+0000 7f079f59e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a00fec20 con 0x7f07a0070430 2024-01-21T19:48:23.792 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.796+0000 7f079e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f07a0070430 0x7f07a00fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:23.792 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.796+0000 7f079e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f07a0070430 0x7f07a00fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47266/0 (socket says 172.21.15.169:47266) 2024-01-21T19:48:23.792 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.796+0000 7f079e59c700 1 -- 172.21.15.169:0/106291370 learned_addr learned my addr 172.21.15.169:0/106291370 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:23.792 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.796+0000 7f079e59c700 1 -- 172.21.15.169:0/106291370 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a00fed60 con 0x7f07a0070430 2024-01-21T19:48:23.792 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.796+0000 7f079e59c700 1 --2- 172.21.15.169:0/106291370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f07a0070430 0x7f07a00fe730 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f079000b5f0 tx=0x7f0790006ee0).ready entity=mon.0 client_cookie=d24ec1fa980aaa1f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:23.792 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.796+0000 7f079d59a700 1 -- 172.21.15.169:0/106291370 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f079000d600 con 0x7f07a0070430 2024-01-21T19:48:23.792 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.796+0000 7f079d59a700 1 -- 172.21.15.169:0/106291370 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f079000dc10 con 0x7f07a0070430 2024-01-21T19:48:23.792 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.796+0000 7f079d59a700 1 -- 172.21.15.169:0/106291370 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0790011cd0 con 0x7f07a0070430 2024-01-21T19:48:23.792 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.797+0000 7f079f59e700 1 -- 172.21.15.169:0/106291370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f07a0070430 msgr2=0x7f07a00fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:23.792 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.797+0000 7f079f59e700 1 --2- 172.21.15.169:0/106291370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f07a0070430 0x7f07a00fe730 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f079000b5f0 tx=0x7f0790006ee0).stop 2024-01-21T19:48:23.792 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.797+0000 7f079f59e700 1 -- 172.21.15.169:0/106291370 shutdown_connections 2024-01-21T19:48:23.792 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.797+0000 7f079f59e700 1 --2- 172.21.15.169:0/106291370 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f07a0070430 0x7f07a00fe730 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:23.792 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.797+0000 7f079f59e700 1 -- 172.21.15.169:0/106291370 >> 172.21.15.169:0/106291370 conn(0x7f07a00f6210 msgr2=0x7f07a00f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:23.793 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.797+0000 7f079f59e700 1 -- 172.21.15.169:0/106291370 shutdown_connections 2024-01-21T19:48:23.793 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.797+0000 7f079f59e700 1 -- 172.21.15.169:0/106291370 wait complete. 2024-01-21T19:48:23.793 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.797+0000 7f079f59e700 1 Processor -- start 2024-01-21T19:48:23.793 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.798+0000 7f079f59e700 1 -- start start 2024-01-21T19:48:23.793 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.798+0000 7f079f59e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f07a0070430 0x7f07a010a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:23.793 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.798+0000 7f079f59e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a010ad80 con 0x7f07a0070430 2024-01-21T19:48:23.793 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.798+0000 7f079e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f07a0070430 0x7f07a010a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:23.793 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.798+0000 7f079e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f07a0070430 0x7f07a010a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47282/0 (socket says 172.21.15.169:47282) 2024-01-21T19:48:23.793 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.798+0000 7f079e59c700 1 -- 172.21.15.169:0/2404743967 learned_addr learned my addr 172.21.15.169:0/2404743967 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:23.794 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.798+0000 7f079e59c700 1 -- 172.21.15.169:0/2404743967 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079000b200 con 0x7f07a0070430 2024-01-21T19:48:23.794 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.798+0000 7f079e59c700 1 --2- 172.21.15.169:0/2404743967 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f07a0070430 0x7f07a010a890 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f0790006a40 tx=0x7f0790006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:23.794 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.798+0000 7f0796ffd700 1 -- 172.21.15.169:0/2404743967 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0790011c90 con 0x7f07a0070430 2024-01-21T19:48:23.794 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.798+0000 7f0796ffd700 1 -- 172.21.15.169:0/2404743967 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f079001b8c0 con 0x7f07a0070430 2024-01-21T19:48:23.794 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.798+0000 7f079f59e700 1 -- 172.21.15.169:0/2404743967 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a010af80 con 0x7f07a0070430 2024-01-21T19:48:23.794 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.798+0000 7f0796ffd700 1 -- 172.21.15.169:0/2404743967 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f079001a8c0 con 0x7f07a0070430 2024-01-21T19:48:23.795 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.798+0000 7f079f59e700 1 -- 172.21.15.169:0/2404743967 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a010b390 con 0x7f07a0070430 2024-01-21T19:48:23.795 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.799+0000 7f0796ffd700 1 -- 172.21.15.169:0/2404743967 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f079000f070 con 0x7f07a0070430 2024-01-21T19:48:23.795 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.799+0000 7f0796ffd700 1 --2- 172.21.15.169:0/2404743967 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f078002fe60 0x7f0780032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:23.795 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.799+0000 7f0796ffd700 1 -- 172.21.15.169:0/2404743967 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f0790040a00 con 0x7f07a0070430 2024-01-21T19:48:23.795 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.799+0000 7f079f59e700 1 -- 172.21.15.169:0/2404743967 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a0045ad0 con 0x7f07a0070430 2024-01-21T19:48:23.795 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.799+0000 7f079dd9b700 1 --2- 172.21.15.169:0/2404743967 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f078002fe60 0x7f0780032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:23.795 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.800+0000 7f079dd9b700 1 --2- 172.21.15.169:0/2404743967 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f078002fe60 0x7f0780032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f07880096a0 tx=0x7f0788006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:23.796 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.803+0000 7f0796ffd700 1 -- 172.21.15.169:0/2404743967 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0790010930 con 0x7f07a0070430 2024-01-21T19:48:23.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.953+0000 7f079f59e700 1 -- 172.21.15.169:0/2404743967 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f07a00ffa50 con 0x7f07a0070430 2024-01-21T19:48:23.948 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.955+0000 7f0796ffd700 1 -- 172.21.15.169:0/2404743967 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v9)=0 v9) v1 ==== 142+0+0 (secure 0 0 0) 0x7f079002f0d0 con 0x7f07a0070430 2024-01-21T19:48:23.952 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.957+0000 7f079f59e700 1 -- 172.21.15.169:0/2404743967 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f078002fe60 msgr2=0x7f0780032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:23.952 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.957+0000 7f079f59e700 1 --2- 172.21.15.169:0/2404743967 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f078002fe60 0x7f0780032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f07880096a0 tx=0x7f0788006b40).stop 2024-01-21T19:48:23.952 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.957+0000 7f079f59e700 1 -- 172.21.15.169:0/2404743967 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f07a0070430 msgr2=0x7f07a010a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:23.952 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.957+0000 7f079f59e700 1 --2- 172.21.15.169:0/2404743967 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f07a0070430 0x7f07a010a890 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f0790006a40 tx=0x7f0790006ee0).stop 2024-01-21T19:48:23.952 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.958+0000 7f079f59e700 1 -- 172.21.15.169:0/2404743967 shutdown_connections 2024-01-21T19:48:23.952 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.958+0000 7f079f59e700 1 --2- 172.21.15.169:0/2404743967 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f078002fe60 0x7f0780032320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:23.952 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.958+0000 7f079f59e700 1 --2- 172.21.15.169:0/2404743967 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f07a0070430 0x7f07a010a890 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:23.952 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.958+0000 7f079f59e700 1 -- 172.21.15.169:0/2404743967 >> 172.21.15.169:0/2404743967 conn(0x7f07a00f6210 msgr2=0x7f07a00f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:23.952 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.958+0000 7f079f59e700 1 -- 172.21.15.169:0/2404743967 shutdown_connections 2024-01-21T19:48:23.952 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:23.958+0000 7f079f59e700 1 -- 172.21.15.169:0/2404743967 wait complete. 2024-01-21T19:48:24.253 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:24 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:22.133706+0000 mgr.smithi169.espgue (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:48:24.253 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:24 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cephadm 2024-01-21T19:48:22.134335+0000 mgr.smithi169.espgue (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-01-21T19:48:24.253 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:24 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:23.062889+0000 mgr.smithi169.espgue (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:48:24.253 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:24 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cephadm 2024-01-21T19:48:23.063433+0000 mgr.smithi169.espgue (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-01-21T19:48:24.254 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:24 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:23.068403+0000 mon.smithi169 (mon.0) 85 : audit [INF] from='mgr.14142 172.21.15.169:0/778638721' entity='mgr.smithi169.espgue' 2024-01-21T19:48:24.254 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:24 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:23.956736+0000 mon.smithi169 (mon.0) 86 : audit [INF] from='client.? 172.21.15.169:0/2404743967' entity='client.admin' 2024-01-21T19:48:24.317 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-01-21T19:48:24.720 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.723+0000 7fb5b908e700 1 Processor -- start 2024-01-21T19:48:24.720 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.723+0000 7fb5b908e700 1 -- start start 2024-01-21T19:48:24.720 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.723+0000 7fb5b908e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5b40703d0 0x7fb5b4100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:24.720 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.723+0000 7fb5b908e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b40fc730 con 0x7fb5b40703d0 2024-01-21T19:48:24.720 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.723+0000 7fb5b2d9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5b40703d0 0x7fb5b4100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:24.721 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.723+0000 7fb5b2d9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5b40703d0 0x7fb5b4100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47284/0 (socket says 172.21.15.169:47284) 2024-01-21T19:48:24.721 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.723+0000 7fb5b2d9d700 1 -- 172.21.15.169:0/1312878162 learned_addr learned my addr 172.21.15.169:0/1312878162 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:24.721 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.723+0000 7fb5b2d9d700 1 -- 172.21.15.169:0/1312878162 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b4101230 con 0x7fb5b40703d0 2024-01-21T19:48:24.721 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.724+0000 7fb5b2d9d700 1 --2- 172.21.15.169:0/1312878162 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5b40703d0 0x7fb5b4100d40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb5a400b5f0 tx=0x7fb5a4006ee0).ready entity=mon.0 client_cookie=26ba5b180e7c47f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:24.721 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.724+0000 7fb5b1d9b700 1 -- 172.21.15.169:0/1312878162 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5a400d600 con 0x7fb5b40703d0 2024-01-21T19:48:24.721 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.724+0000 7fb5b1d9b700 1 -- 172.21.15.169:0/1312878162 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb5a400dc10 con 0x7fb5b40703d0 2024-01-21T19:48:24.721 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.724+0000 7fb5b1d9b700 1 -- 172.21.15.169:0/1312878162 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5a4011cd0 con 0x7fb5b40703d0 2024-01-21T19:48:24.721 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.724+0000 7fb5b908e700 1 -- 172.21.15.169:0/1312878162 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5b40703d0 msgr2=0x7fb5b4100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:24.721 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.724+0000 7fb5b908e700 1 --2- 172.21.15.169:0/1312878162 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5b40703d0 0x7fb5b4100d40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb5a400b5f0 tx=0x7fb5a4006ee0).stop 2024-01-21T19:48:24.721 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.724+0000 7fb5b908e700 1 -- 172.21.15.169:0/1312878162 shutdown_connections 2024-01-21T19:48:24.721 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.724+0000 7fb5b908e700 1 --2- 172.21.15.169:0/1312878162 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5b40703d0 0x7fb5b4100d40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:24.721 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.724+0000 7fb5b908e700 1 -- 172.21.15.169:0/1312878162 >> 172.21.15.169:0/1312878162 conn(0x7fb5b40f63c0 msgr2=0x7fb5b40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:24.721 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.724+0000 7fb5b908e700 1 -- 172.21.15.169:0/1312878162 shutdown_connections 2024-01-21T19:48:24.721 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.724+0000 7fb5b908e700 1 -- 172.21.15.169:0/1312878162 wait complete. 2024-01-21T19:48:24.721 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.725+0000 7fb5b908e700 1 Processor -- start 2024-01-21T19:48:24.722 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.725+0000 7fb5b908e700 1 -- start start 2024-01-21T19:48:24.722 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.725+0000 7fb5b908e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5b40703d0 0x7fb5b410cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:24.722 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.725+0000 7fb5b908e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b410d1f0 con 0x7fb5b40703d0 2024-01-21T19:48:24.722 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.725+0000 7fb5b2d9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5b40703d0 0x7fb5b410cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:24.722 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.725+0000 7fb5b2d9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5b40703d0 0x7fb5b410cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47292/0 (socket says 172.21.15.169:47292) 2024-01-21T19:48:24.722 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.725+0000 7fb5b2d9d700 1 -- 172.21.15.169:0/1921437643 learned_addr learned my addr 172.21.15.169:0/1921437643 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:24.722 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.725+0000 7fb5b2d9d700 1 -- 172.21.15.169:0/1921437643 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a400b200 con 0x7fb5b40703d0 2024-01-21T19:48:24.722 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.726+0000 7fb5b2d9d700 1 --2- 172.21.15.169:0/1921437643 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5b40703d0 0x7fb5b410cd00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fb5a4006a40 tx=0x7fb5a4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:24.722 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.726+0000 7fb59b7fe700 1 -- 172.21.15.169:0/1921437643 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5a4011c90 con 0x7fb5b40703d0 2024-01-21T19:48:24.722 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.726+0000 7fb5b908e700 1 -- 172.21.15.169:0/1921437643 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5b410d3f0 con 0x7fb5b40703d0 2024-01-21T19:48:24.722 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.726+0000 7fb59b7fe700 1 -- 172.21.15.169:0/1921437643 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb5a401b8c0 con 0x7fb5b40703d0 2024-01-21T19:48:24.722 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.726+0000 7fb59b7fe700 1 -- 172.21.15.169:0/1921437643 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5a401a8c0 con 0x7fb5b40703d0 2024-01-21T19:48:24.722 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.726+0000 7fb5b908e700 1 -- 172.21.15.169:0/1921437643 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b410d800 con 0x7fb5b40703d0 2024-01-21T19:48:24.722 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.727+0000 7fb59b7fe700 1 -- 172.21.15.169:0/1921437643 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fb5a400f070 con 0x7fb5b40703d0 2024-01-21T19:48:24.722 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.727+0000 7fb5b908e700 1 -- 172.21.15.169:0/1921437643 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5b4045ad0 con 0x7fb5b40703d0 2024-01-21T19:48:24.723 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.727+0000 7fb59b7fe700 1 --2- 172.21.15.169:0/1921437643 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fb5a002fe60 0x7fb5a0032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:24.723 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.727+0000 7fb59b7fe700 1 -- 172.21.15.169:0/1921437643 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fb5a4040a20 con 0x7fb5b40703d0 2024-01-21T19:48:24.723 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.727+0000 7fb5b259c700 1 --2- 172.21.15.169:0/1921437643 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fb5a002fe60 0x7fb5a0032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:24.723 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.727+0000 7fb5b259c700 1 --2- 172.21.15.169:0/1921437643 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fb5a002fe60 0x7fb5a0032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fb5a80096a0 tx=0x7fb5a8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:24.724 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.731+0000 7fb59b7fe700 1 -- 172.21.15.169:0/1921437643 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb5a4010900 con 0x7fb5b40703d0 2024-01-21T19:48:24.875 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.881+0000 7fb5b908e700 1 -- 172.21.15.169:0/1921437643 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fb5b4101e00 con 0x7fb5b40703d0 2024-01-21T19:48:24.877 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.884+0000 7fb59b7fe700 1 -- 172.21.15.169:0/1921437643 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7fb5a4016070 con 0x7fb5b40703d0 2024-01-21T19:48:24.880 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.886+0000 7fb5b908e700 1 -- 172.21.15.169:0/1921437643 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fb5a002fe60 msgr2=0x7fb5a0032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:24.880 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.886+0000 7fb5b908e700 1 --2- 172.21.15.169:0/1921437643 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fb5a002fe60 0x7fb5a0032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fb5a80096a0 tx=0x7fb5a8006b40).stop 2024-01-21T19:48:24.881 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.886+0000 7fb5b908e700 1 -- 172.21.15.169:0/1921437643 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5b40703d0 msgr2=0x7fb5b410cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:24.881 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.886+0000 7fb5b908e700 1 --2- 172.21.15.169:0/1921437643 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5b40703d0 0x7fb5b410cd00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fb5a4006a40 tx=0x7fb5a4006ee0).stop 2024-01-21T19:48:24.881 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.886+0000 7fb5b908e700 1 -- 172.21.15.169:0/1921437643 shutdown_connections 2024-01-21T19:48:24.881 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.886+0000 7fb5b908e700 1 --2- 172.21.15.169:0/1921437643 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fb5a002fe60 0x7fb5a0032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:24.881 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.886+0000 7fb5b908e700 1 --2- 172.21.15.169:0/1921437643 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5b40703d0 0x7fb5b410cd00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:24.881 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.886+0000 7fb5b908e700 1 -- 172.21.15.169:0/1921437643 >> 172.21.15.169:0/1921437643 conn(0x7fb5b40f63c0 msgr2=0x7fb5b40f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:24.881 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.886+0000 7fb5b908e700 1 -- 172.21.15.169:0/1921437643 shutdown_connections 2024-01-21T19:48:24.881 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:24.886+0000 7fb5b908e700 1 -- 172.21.15.169:0/1921437643 wait complete. 2024-01-21T19:48:25.235 INFO:teuthology.orchestra.run.smithi169.stderr:Enabling the dashboard module... 2024-01-21T19:48:25.235 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-01-21T19:48:25.585 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.587+0000 7f7b39d38700 1 Processor -- start 2024-01-21T19:48:25.585 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.587+0000 7f7b39d38700 1 -- start start 2024-01-21T19:48:25.585 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.588+0000 7f7b39d38700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b34070430 0x7f7b340fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:25.585 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.588+0000 7f7b39d38700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b340fedd0 con 0x7f7b34070430 2024-01-21T19:48:25.585 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.588+0000 7f7b337fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b34070430 0x7f7b340fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:25.585 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.588+0000 7f7b337fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b34070430 0x7f7b340fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47296/0 (socket says 172.21.15.169:47296) 2024-01-21T19:48:25.585 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.588+0000 7f7b337fe700 1 -- 172.21.15.169:0/2780448575 learned_addr learned my addr 172.21.15.169:0/2780448575 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:25.585 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.588+0000 7f7b337fe700 1 -- 172.21.15.169:0/2780448575 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b340fef10 con 0x7f7b34070430 2024-01-21T19:48:25.585 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.588+0000 7f7b337fe700 1 --2- 172.21.15.169:0/2780448575 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b34070430 0x7f7b340fe8e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f7b1c00b5f0 tx=0x7f7b1c006ee0).ready entity=mon.0 client_cookie=fc839d1d5775582e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:25.585 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.588+0000 7f7b327fc700 1 -- 172.21.15.169:0/2780448575 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b1c00d600 con 0x7f7b34070430 2024-01-21T19:48:25.585 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.588+0000 7f7b327fc700 1 -- 172.21.15.169:0/2780448575 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b1c00dc10 con 0x7f7b34070430 2024-01-21T19:48:25.585 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.588+0000 7f7b327fc700 1 -- 172.21.15.169:0/2780448575 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b1c011cd0 con 0x7f7b34070430 2024-01-21T19:48:25.585 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.589+0000 7f7b39d38700 1 -- 172.21.15.169:0/2780448575 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b34070430 msgr2=0x7f7b340fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:25.586 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.589+0000 7f7b39d38700 1 --2- 172.21.15.169:0/2780448575 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b34070430 0x7f7b340fe8e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f7b1c00b5f0 tx=0x7f7b1c006ee0).stop 2024-01-21T19:48:25.586 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.589+0000 7f7b39d38700 1 -- 172.21.15.169:0/2780448575 shutdown_connections 2024-01-21T19:48:25.586 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.589+0000 7f7b39d38700 1 --2- 172.21.15.169:0/2780448575 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b34070430 0x7f7b340fe8e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:25.586 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.589+0000 7f7b39d38700 1 -- 172.21.15.169:0/2780448575 >> 172.21.15.169:0/2780448575 conn(0x7f7b340f63c0 msgr2=0x7f7b340f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:25.586 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.589+0000 7f7b39d38700 1 -- 172.21.15.169:0/2780448575 shutdown_connections 2024-01-21T19:48:25.586 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.589+0000 7f7b39d38700 1 -- 172.21.15.169:0/2780448575 wait complete. 2024-01-21T19:48:25.586 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.590+0000 7f7b39d38700 1 Processor -- start 2024-01-21T19:48:25.586 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.590+0000 7f7b39d38700 1 -- start start 2024-01-21T19:48:25.586 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.590+0000 7f7b39d38700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b34070430 0x7f7b340fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:25.586 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.590+0000 7f7b39d38700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b340fa7e0 con 0x7f7b34070430 2024-01-21T19:48:25.586 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.590+0000 7f7b337fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b34070430 0x7f7b340fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:25.586 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.590+0000 7f7b337fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b34070430 0x7f7b340fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47298/0 (socket says 172.21.15.169:47298) 2024-01-21T19:48:25.586 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.590+0000 7f7b337fe700 1 -- 172.21.15.169:0/2014948791 learned_addr learned my addr 172.21.15.169:0/2014948791 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:25.586 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.590+0000 7f7b337fe700 1 -- 172.21.15.169:0/2014948791 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b1c00b200 con 0x7f7b34070430 2024-01-21T19:48:25.586 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.590+0000 7f7b337fe700 1 --2- 172.21.15.169:0/2014948791 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b34070430 0x7f7b340fc360 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f7b1c006800 tx=0x7f7b1c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:25.586 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.591+0000 7f7b2bfff700 1 -- 172.21.15.169:0/2014948791 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b1c011df0 con 0x7f7b34070430 2024-01-21T19:48:25.586 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.591+0000 7f7b2bfff700 1 -- 172.21.15.169:0/2014948791 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b1c01b8c0 con 0x7f7b34070430 2024-01-21T19:48:25.587 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.591+0000 7f7b2bfff700 1 -- 172.21.15.169:0/2014948791 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b1c01a830 con 0x7f7b34070430 2024-01-21T19:48:25.587 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.591+0000 7f7b39d38700 1 -- 172.21.15.169:0/2014948791 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b340fa980 con 0x7f7b34070430 2024-01-21T19:48:25.587 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.591+0000 7f7b39d38700 1 -- 172.21.15.169:0/2014948791 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b340fad90 con 0x7f7b34070430 2024-01-21T19:48:25.587 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.591+0000 7f7b2bfff700 1 -- 172.21.15.169:0/2014948791 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f7b1c00f070 con 0x7f7b34070430 2024-01-21T19:48:25.587 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.592+0000 7f7b2bfff700 1 --2- 172.21.15.169:0/2014948791 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f7b1402fe60 0x7f7b14032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:25.587 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.592+0000 7f7b2bfff700 1 -- 172.21.15.169:0/2014948791 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f7b1c041040 con 0x7f7b34070430 2024-01-21T19:48:25.587 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.592+0000 7f7b32ffd700 1 --2- 172.21.15.169:0/2014948791 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f7b1402fe60 0x7f7b14032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:25.588 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.592+0000 7f7b39d38700 1 -- 172.21.15.169:0/2014948791 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b34059100 con 0x7f7b34070430 2024-01-21T19:48:25.588 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.592+0000 7f7b32ffd700 1 --2- 172.21.15.169:0/2014948791 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f7b1402fe60 0x7f7b14032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f7b240096a0 tx=0x7f7b24006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:25.589 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.596+0000 7f7b2bfff700 1 -- 172.21.15.169:0/2014948791 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b1c016080 con 0x7f7b34070430 2024-01-21T19:48:25.778 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.784+0000 7f7b39d38700 1 -- 172.21.15.169:0/2014948791 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f7b34045ad0 con 0x7f7b34070430 2024-01-21T19:48:25.878 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.884+0000 7f7b2bfff700 1 -- 172.21.15.169:0/2014948791 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f7b1c01a9d0 con 0x7f7b34070430 2024-01-21T19:48:25.879 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.886+0000 7f7b2bfff700 1 -- 172.21.15.169:0/2014948791 <== mon.0 v2:172.21.15.169:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f7b1c02c560 con 0x7f7b34070430 2024-01-21T19:48:25.882 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.888+0000 7f7b39d38700 1 -- 172.21.15.169:0/2014948791 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f7b1402fe60 msgr2=0x7f7b14032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:25.882 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.888+0000 7f7b39d38700 1 --2- 172.21.15.169:0/2014948791 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f7b1402fe60 0x7f7b14032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f7b240096a0 tx=0x7f7b24006b40).stop 2024-01-21T19:48:25.882 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.888+0000 7f7b39d38700 1 -- 172.21.15.169:0/2014948791 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b34070430 msgr2=0x7f7b340fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:25.882 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.888+0000 7f7b39d38700 1 --2- 172.21.15.169:0/2014948791 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b34070430 0x7f7b340fc360 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f7b1c006800 tx=0x7f7b1c006ee0).stop 2024-01-21T19:48:25.882 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.888+0000 7f7b39d38700 1 -- 172.21.15.169:0/2014948791 shutdown_connections 2024-01-21T19:48:25.882 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.888+0000 7f7b39d38700 1 --2- 172.21.15.169:0/2014948791 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f7b1402fe60 0x7f7b14032320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:25.882 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.888+0000 7f7b39d38700 1 --2- 172.21.15.169:0/2014948791 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7b34070430 0x7f7b340fc360 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:25.882 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.888+0000 7f7b39d38700 1 -- 172.21.15.169:0/2014948791 >> 172.21.15.169:0/2014948791 conn(0x7f7b340f63c0 msgr2=0x7f7b340f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:25.882 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.888+0000 7f7b39d38700 1 -- 172.21.15.169:0/2014948791 shutdown_connections 2024-01-21T19:48:25.882 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:25.888+0000 7f7b39d38700 1 -- 172.21.15.169:0/2014948791 wait complete. 2024-01-21T19:48:26.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:25 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:24.885708+0000 mon.smithi169 (mon.0) 87 : audit [INF] from='client.? 172.21.15.169:0/1921437643' entity='client.admin' 2024-01-21T19:48:26.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:25 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:25.785841+0000 mon.smithi169 (mon.0) 88 : audit [INF] from='client.? 172.21.15.169:0/2014948791' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-21T19:48:26.244 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-21T19:48:26.637 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.641+0000 7fc8fae44700 1 Processor -- start 2024-01-21T19:48:26.637 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.641+0000 7fc8fae44700 1 -- start start 2024-01-21T19:48:26.637 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.641+0000 7fc8fae44700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8f40703f0 0x7fc8f40fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:26.637 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.641+0000 7fc8fae44700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f40fee10 con 0x7fc8f40703f0 2024-01-21T19:48:26.637 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.641+0000 7fc8f8be0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8f40703f0 0x7fc8f40fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:26.637 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.641+0000 7fc8f8be0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8f40703f0 0x7fc8f40fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47336/0 (socket says 172.21.15.169:47336) 2024-01-21T19:48:26.637 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.641+0000 7fc8f8be0700 1 -- 172.21.15.169:0/1850894342 learned_addr learned my addr 172.21.15.169:0/1850894342 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:26.637 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.642+0000 7fc8f8be0700 1 -- 172.21.15.169:0/1850894342 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f40fef50 con 0x7fc8f40703f0 2024-01-21T19:48:26.637 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.642+0000 7fc8f8be0700 1 --2- 172.21.15.169:0/1850894342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8f40703f0 0x7fc8f40fe920 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fc8e800b5f0 tx=0x7fc8e8006ee0).ready entity=mon.0 client_cookie=bb4e30c0a78fa81c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:26.637 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.642+0000 7fc8f37fe700 1 -- 172.21.15.169:0/1850894342 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8e800d600 con 0x7fc8f40703f0 2024-01-21T19:48:26.637 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.642+0000 7fc8f37fe700 1 -- 172.21.15.169:0/1850894342 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc8e800dc10 con 0x7fc8f40703f0 2024-01-21T19:48:26.638 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.642+0000 7fc8f37fe700 1 -- 172.21.15.169:0/1850894342 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8e8011cd0 con 0x7fc8f40703f0 2024-01-21T19:48:26.638 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.643+0000 7fc8fae44700 1 -- 172.21.15.169:0/1850894342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8f40703f0 msgr2=0x7fc8f40fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:26.638 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.643+0000 7fc8fae44700 1 --2- 172.21.15.169:0/1850894342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8f40703f0 0x7fc8f40fe920 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fc8e800b5f0 tx=0x7fc8e8006ee0).stop 2024-01-21T19:48:26.638 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.643+0000 7fc8fae44700 1 -- 172.21.15.169:0/1850894342 shutdown_connections 2024-01-21T19:48:26.638 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.643+0000 7fc8fae44700 1 --2- 172.21.15.169:0/1850894342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8f40703f0 0x7fc8f40fe920 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:26.638 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.643+0000 7fc8fae44700 1 -- 172.21.15.169:0/1850894342 >> 172.21.15.169:0/1850894342 conn(0x7fc8f40f63c0 msgr2=0x7fc8f40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:26.638 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.643+0000 7fc8fae44700 1 -- 172.21.15.169:0/1850894342 shutdown_connections 2024-01-21T19:48:26.638 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.643+0000 7fc8fae44700 1 -- 172.21.15.169:0/1850894342 wait complete. 2024-01-21T19:48:26.638 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.643+0000 7fc8fae44700 1 Processor -- start 2024-01-21T19:48:26.640 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.644+0000 7fc8fae44700 1 -- start start 2024-01-21T19:48:26.640 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.644+0000 7fc8fae44700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8f40703f0 0x7fc8f410aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:26.640 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.644+0000 7fc8fae44700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f410afc0 con 0x7fc8f40703f0 2024-01-21T19:48:26.640 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.644+0000 7fc8f8be0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8f40703f0 0x7fc8f410aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:26.640 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.644+0000 7fc8f8be0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8f40703f0 0x7fc8f410aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47348/0 (socket says 172.21.15.169:47348) 2024-01-21T19:48:26.640 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.644+0000 7fc8f8be0700 1 -- 172.21.15.169:0/566120206 learned_addr learned my addr 172.21.15.169:0/566120206 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:26.640 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.644+0000 7fc8f8be0700 1 -- 172.21.15.169:0/566120206 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e800b200 con 0x7fc8f40703f0 2024-01-21T19:48:26.641 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.644+0000 7fc8f8be0700 1 --2- 172.21.15.169:0/566120206 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8f40703f0 0x7fc8f410aad0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fc8e800df90 tx=0x7fc8e8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:26.641 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.645+0000 7fc8f17fa700 1 -- 172.21.15.169:0/566120206 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8e8011da0 con 0x7fc8f40703f0 2024-01-21T19:48:26.641 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.645+0000 7fc8f17fa700 1 -- 172.21.15.169:0/566120206 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc8e801b8c0 con 0x7fc8f40703f0 2024-01-21T19:48:26.641 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.645+0000 7fc8fae44700 1 -- 172.21.15.169:0/566120206 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f410b1c0 con 0x7fc8f40703f0 2024-01-21T19:48:26.641 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.645+0000 7fc8f17fa700 1 -- 172.21.15.169:0/566120206 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8e801a930 con 0x7fc8f40703f0 2024-01-21T19:48:26.641 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.645+0000 7fc8fae44700 1 -- 172.21.15.169:0/566120206 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f410b5d0 con 0x7fc8f40703f0 2024-01-21T19:48:26.641 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.645+0000 7fc8f17fa700 1 -- 172.21.15.169:0/566120206 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fc8e800f070 con 0x7fc8f40703f0 2024-01-21T19:48:26.641 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.646+0000 7fc8f17fa700 1 --2- 172.21.15.169:0/566120206 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fc8d802feb0 0x7fc8d8032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:26.641 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.646+0000 7fc8f17fa700 1 -- 172.21.15.169:0/566120206 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fc8e8042040 con 0x7fc8f40703f0 2024-01-21T19:48:26.641 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.646+0000 7fc8f3fff700 1 -- 172.21.15.169:0/566120206 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fc8d802feb0 msgr2=0x7fc8d8032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/528052065 2024-01-21T19:48:26.641 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.646+0000 7fc8f3fff700 1 --2- 172.21.15.169:0/566120206 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fc8d802feb0 0x7fc8d8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T19:48:26.641 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.646+0000 7fc8fae44700 1 -- 172.21.15.169:0/566120206 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f4059100 con 0x7fc8f40703f0 2024-01-21T19:48:26.643 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.650+0000 7fc8f17fa700 1 -- 172.21.15.169:0/566120206 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc8e8016070 con 0x7fc8f40703f0 2024-01-21T19:48:26.834 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.840+0000 7fc8fae44700 1 -- 172.21.15.169:0/566120206 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fc8f40ffc40 con 0x7fc8f40703f0 2024-01-21T19:48:26.835 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.841+0000 7fc8f17fa700 1 -- 172.21.15.169:0/566120206 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7fc8e8010be0 con 0x7fc8f40703f0 2024-01-21T19:48:26.835 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: { 2024-01-21T19:48:26.835 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "epoch": 13, 2024-01-21T19:48:26.835 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "available": true, 2024-01-21T19:48:26.835 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "active_name": "smithi169.espgue", 2024-01-21T19:48:26.835 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-21T19:48:26.835 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: } 2024-01-21T19:48:26.836 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.842+0000 7fc8fae44700 1 -- 172.21.15.169:0/566120206 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fc8d802feb0 msgr2=0x7fc8d8032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T19:48:26.837 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.842+0000 7fc8fae44700 1 --2- 172.21.15.169:0/566120206 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fc8d802feb0 0x7fc8d8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:48:26.837 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.842+0000 7fc8fae44700 1 -- 172.21.15.169:0/566120206 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8f40703f0 msgr2=0x7fc8f410aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:26.837 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.842+0000 7fc8fae44700 1 --2- 172.21.15.169:0/566120206 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8f40703f0 0x7fc8f410aad0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fc8e800df90 tx=0x7fc8e8006ee0).stop 2024-01-21T19:48:26.837 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.843+0000 7fc8fae44700 1 -- 172.21.15.169:0/566120206 shutdown_connections 2024-01-21T19:48:26.837 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.843+0000 7fc8fae44700 1 --2- 172.21.15.169:0/566120206 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7fc8d802feb0 0x7fc8d8032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:48:26.837 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.843+0000 7fc8fae44700 1 --2- 172.21.15.169:0/566120206 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc8f40703f0 0x7fc8f410aad0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:26.837 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.843+0000 7fc8fae44700 1 -- 172.21.15.169:0/566120206 >> 172.21.15.169:0/566120206 conn(0x7fc8f40f63c0 msgr2=0x7fc8f40f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:26.837 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.843+0000 7fc8fae44700 1 -- 172.21.15.169:0/566120206 shutdown_connections 2024-01-21T19:48:26.837 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:26.843+0000 7fc8fae44700 1 -- 172.21.15.169:0/566120206 wait complete. 2024-01-21T19:48:27.148 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:26 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:25.886971+0000 mon.smithi169 (mon.0) 89 : audit [INF] from='client.? 172.21.15.169:0/2014948791' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-21T19:48:27.148 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:26 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:25.887022+0000 mon.smithi169 (mon.0) 90 : cluster [DBG] mgrmap e13: smithi169.espgue(active, since 7s) 2024-01-21T19:48:27.148 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:26 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:26.841931+0000 mon.smithi169 (mon.0) 91 : audit [DBG] from='client.? 172.21.15.169:0/566120206' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-21T19:48:27.209 INFO:teuthology.orchestra.run.smithi169.stderr:Waiting for the mgr to restart... 2024-01-21T19:48:27.210 INFO:teuthology.orchestra.run.smithi169.stderr:Waiting for mgr epoch 13... 2024-01-21T19:48:27.210 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-21T19:48:27.550 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.556+0000 7f56e86be700 1 Processor -- start 2024-01-21T19:48:27.551 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.556+0000 7f56e86be700 1 -- start start 2024-01-21T19:48:27.551 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.556+0000 7f56e86be700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56e0070370 0x7f56e00fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:27.551 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.556+0000 7f56e86be700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e0070790 con 0x7f56e0070370 2024-01-21T19:48:27.551 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.556+0000 7f56e645a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56e0070370 0x7f56e00fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:27.551 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.556+0000 7f56e645a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56e0070370 0x7f56e00fe9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47356/0 (socket says 172.21.15.169:47356) 2024-01-21T19:48:27.551 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.556+0000 7f56e645a700 1 -- 172.21.15.169:0/1303420735 learned_addr learned my addr 172.21.15.169:0/1303420735 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:27.551 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.557+0000 7f56e645a700 1 -- 172.21.15.169:0/1303420735 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e00fee90 con 0x7f56e0070370 2024-01-21T19:48:27.551 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.557+0000 7f56e645a700 1 --2- 172.21.15.169:0/1303420735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56e0070370 0x7f56e00fe9a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f56dc00b5f0 tx=0x7f56dc006ee0).ready entity=mon.0 client_cookie=af70df30f8d95850 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:27.551 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.557+0000 7f56e5458700 1 -- 172.21.15.169:0/1303420735 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56dc00d600 con 0x7f56e0070370 2024-01-21T19:48:27.551 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.557+0000 7f56e5458700 1 -- 172.21.15.169:0/1303420735 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f56dc00dc10 con 0x7f56e0070370 2024-01-21T19:48:27.551 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.557+0000 7f56e5458700 1 -- 172.21.15.169:0/1303420735 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56dc011cd0 con 0x7f56e0070370 2024-01-21T19:48:27.554 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.558+0000 7f56e86be700 1 -- 172.21.15.169:0/1303420735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56e0070370 msgr2=0x7f56e00fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:27.554 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.558+0000 7f56e86be700 1 --2- 172.21.15.169:0/1303420735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56e0070370 0x7f56e00fe9a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f56dc00b5f0 tx=0x7f56dc006ee0).stop 2024-01-21T19:48:27.554 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.558+0000 7f56e86be700 1 -- 172.21.15.169:0/1303420735 shutdown_connections 2024-01-21T19:48:27.555 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.558+0000 7f56e86be700 1 --2- 172.21.15.169:0/1303420735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56e0070370 0x7f56e00fe9a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:27.555 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.558+0000 7f56e86be700 1 -- 172.21.15.169:0/1303420735 >> 172.21.15.169:0/1303420735 conn(0x7f56e00f63c0 msgr2=0x7f56e00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:27.555 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.558+0000 7f56e86be700 1 -- 172.21.15.169:0/1303420735 shutdown_connections 2024-01-21T19:48:27.555 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.558+0000 7f56e86be700 1 -- 172.21.15.169:0/1303420735 wait complete. 2024-01-21T19:48:27.555 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.559+0000 7f56e86be700 1 Processor -- start 2024-01-21T19:48:27.555 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.559+0000 7f56e86be700 1 -- start start 2024-01-21T19:48:27.555 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.559+0000 7f56e86be700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56e0070370 0x7f56e00fc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:27.555 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.559+0000 7f56e86be700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e00fa8a0 con 0x7f56e0070370 2024-01-21T19:48:27.555 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.559+0000 7f56e645a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56e0070370 0x7f56e00fc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:27.555 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.559+0000 7f56e645a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56e0070370 0x7f56e00fc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:47358/0 (socket says 172.21.15.169:47358) 2024-01-21T19:48:27.555 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.559+0000 7f56e645a700 1 -- 172.21.15.169:0/1963125142 learned_addr learned my addr 172.21.15.169:0/1963125142 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:27.555 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.559+0000 7f56e645a700 1 -- 172.21.15.169:0/1963125142 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56dc00b200 con 0x7f56e0070370 2024-01-21T19:48:27.555 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.559+0000 7f56e645a700 1 --2- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56e0070370 0x7f56e00fc420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f56dc006800 tx=0x7f56dc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:27.555 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.560+0000 7f56ceffd700 1 -- 172.21.15.169:0/1963125142 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56dc011df0 con 0x7f56e0070370 2024-01-21T19:48:27.556 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.560+0000 7f56e86be700 1 -- 172.21.15.169:0/1963125142 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e00faa40 con 0x7f56e0070370 2024-01-21T19:48:27.556 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.560+0000 7f56ceffd700 1 -- 172.21.15.169:0/1963125142 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f56dc01b8c0 con 0x7f56e0070370 2024-01-21T19:48:27.556 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.560+0000 7f56ceffd700 1 -- 172.21.15.169:0/1963125142 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56dc01a890 con 0x7f56e0070370 2024-01-21T19:48:27.556 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.560+0000 7f56e86be700 1 -- 172.21.15.169:0/1963125142 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e00fae50 con 0x7f56e0070370 2024-01-21T19:48:27.556 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.560+0000 7f56ceffd700 1 -- 172.21.15.169:0/1963125142 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f56dc00f070 con 0x7f56e0070370 2024-01-21T19:48:27.556 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.561+0000 7f56ceffd700 1 --2- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f56c802ff00 0x7f56c80323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:27.556 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.561+0000 7f56e86be700 1 -- 172.21.15.169:0/1963125142 --> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f56e0059100 con 0x7f56c802ff00 2024-01-21T19:48:27.556 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.561+0000 7f56ceffd700 1 -- 172.21.15.169:0/1963125142 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f56dc040fa0 con 0x7f56e0070370 2024-01-21T19:48:27.556 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.561+0000 7f56e5c59700 1 -- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f56c802ff00 msgr2=0x7f56c80323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/528052065 2024-01-21T19:48:27.556 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.561+0000 7f56e5c59700 1 --2- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f56c802ff00 0x7f56c80323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T19:48:27.755 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.762+0000 7f56e5c59700 1 -- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f56c802ff00 msgr2=0x7f56c80323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/528052065 2024-01-21T19:48:27.755 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:27.762+0000 7f56e5c59700 1 --2- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f56c802ff00 0x7f56c80323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T19:48:28.156 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:28.163+0000 7f56e5c59700 1 -- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f56c802ff00 msgr2=0x7f56c80323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/528052065 2024-01-21T19:48:28.156 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:28.163+0000 7f56e5c59700 1 --2- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f56c802ff00 0x7f56c80323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-21T19:48:28.957 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:28.964+0000 7f56e5c59700 1 -- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f56c802ff00 msgr2=0x7f56c80323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/528052065 2024-01-21T19:48:28.957 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:28.964+0000 7f56e5c59700 1 --2- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f56c802ff00 0x7f56c80323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-21T19:48:30.558 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:30.565+0000 7f56e5c59700 1 -- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f56c802ff00 msgr2=0x7f56c80323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6800/528052065 2024-01-21T19:48:30.558 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:30.565+0000 7f56e5c59700 1 --2- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f56c802ff00 0x7f56c80323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-21T19:48:30.922 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:30.928+0000 7f56ceffd700 1 -- 172.21.15.169:0/1963125142 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f56dc01a9f0 con 0x7f56e0070370 2024-01-21T19:48:30.922 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:30.928+0000 7f56ceffd700 1 -- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f56c802ff00 msgr2=0x7f56c80323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T19:48:30.922 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:30.928+0000 7f56ceffd700 1 --2- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f56c802ff00 0x7f56c80323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:48:31.151 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:30 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:30.876930+0000 mon.smithi169 (mon.0) 92 : cluster [INF] Active manager daemon smithi169.espgue restarted 2024-01-21T19:48:31.151 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:30 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:30.877354+0000 mon.smithi169 (mon.0) 93 : cluster [INF] Activating manager daemon smithi169.espgue 2024-01-21T19:48:31.151 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:30 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:30.879115+0000 mon.smithi169 (mon.0) 94 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-01-21T19:48:31.934 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:31.940+0000 7f56ceffd700 1 -- 172.21.15.169:0/1963125142 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f56dc0420f0 con 0x7f56e0070370 2024-01-21T19:48:31.934 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:31.940+0000 7f56ceffd700 1 --2- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f56c80339e0 0x7f56c8035dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:31.935 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:31.940+0000 7f56ceffd700 1 -- 172.21.15.169:0/1963125142 --> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f56e0059100 con 0x7f56c80339e0 2024-01-21T19:48:31.935 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:31.940+0000 7f56e5c59700 1 --2- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f56c80339e0 0x7f56c8035dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:31.935 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:31.941+0000 7f56e5c59700 1 --2- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f56c80339e0 0x7f56c8035dd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f56d0006fd0 tx=0x7f56d0006d70).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:31.935 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:31.941+0000 7f56ceffd700 1 -- 172.21.15.169:0/1963125142 <== mgr.14162 v2:172.21.15.169:6800/1381810408 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f56e0059100 con 0x7f56c80339e0 2024-01-21T19:48:31.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:31.952+0000 7f56e86be700 1 -- 172.21.15.169:0/1963125142 --> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f56e0045ad0 con 0x7f56c80339e0 2024-01-21T19:48:31.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:31.952+0000 7f56ceffd700 1 -- 172.21.15.169:0/1963125142 <== mgr.14162 v2:172.21.15.169:6800/1381810408 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f56e0045ad0 con 0x7f56c80339e0 2024-01-21T19:48:31.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: { 2024-01-21T19:48:31.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-01-21T19:48:31.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: "initialized": true 2024-01-21T19:48:31.946 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: } 2024-01-21T19:48:31.947 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:31.953+0000 7f56e86be700 1 -- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f56c80339e0 msgr2=0x7f56c8035dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:31.947 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:31.953+0000 7f56e86be700 1 --2- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f56c80339e0 0x7f56c8035dd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f56d0006fd0 tx=0x7f56d0006d70).stop 2024-01-21T19:48:31.948 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:31.953+0000 7f56e86be700 1 -- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56e0070370 msgr2=0x7f56e00fc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:31.948 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:31.953+0000 7f56e86be700 1 --2- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56e0070370 0x7f56e00fc420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f56dc006800 tx=0x7f56dc006ee0).stop 2024-01-21T19:48:31.948 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:31.953+0000 7f56e86be700 1 -- 172.21.15.169:0/1963125142 shutdown_connections 2024-01-21T19:48:31.948 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:31.953+0000 7f56e86be700 1 --2- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f56c80339e0 0x7f56c8035dd0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:31.948 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:31.953+0000 7f56e86be700 1 --2- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56e0070370 0x7f56e00fc420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:31.948 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:31.953+0000 7f56e86be700 1 --2- 172.21.15.169:0/1963125142 >> [v2:172.21.15.169:6800/528052065,v1:172.21.15.169:6801/528052065] conn(0x7f56c802ff00 0x7f56c80323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:48:31.948 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:31.953+0000 7f56e86be700 1 -- 172.21.15.169:0/1963125142 >> 172.21.15.169:0/1963125142 conn(0x7f56e00f63c0 msgr2=0x7f56e00f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:31.948 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:31.953+0000 7f56e86be700 1 -- 172.21.15.169:0/1963125142 shutdown_connections 2024-01-21T19:48:31.948 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:31.953+0000 7f56e86be700 1 -- 172.21.15.169:0/1963125142 wait complete. 2024-01-21T19:48:32.116 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:30.930505+0000 mon.smithi169 (mon.0) 95 : cluster [DBG] mgrmap e14: smithi169.espgue(active, starting, since 0.0532558s) 2024-01-21T19:48:32.116 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:30.931933+0000 mon.smithi169 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-01-21T19:48:32.116 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:30.932506+0000 mon.smithi169 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix": "mgr metadata", "who": "smithi169.espgue", "id": "smithi169.espgue"}]: dispatch 2024-01-21T19:48:32.116 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:30.933133+0000 mon.smithi169 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T19:48:32.116 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:30.933260+0000 mon.smithi169 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T19:48:32.116 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:30.933383+0000 mon.smithi169 (mon.0) 100 : audit [DBG] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T19:48:32.116 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:30.947701+0000 mon.smithi169 (mon.0) 101 : cluster [INF] Manager daemon smithi169.espgue is now available 2024-01-21T19:48:32.117 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:31.044313+0000 mon.smithi169 (mon.0) 102 : audit [DBG] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:48:32.117 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:31.046456+0000 mon.smithi169 (mon.0) 103 : audit [DBG] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:48:32.117 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:31.063662+0000 mon.smithi169 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/mirror_snapshot_schedule"}]: dispatch 2024-01-21T19:48:32.117 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:31.071619+0000 mon.smithi169 (mon.0) 105 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/trash_purge_schedule"}]: dispatch 2024-01-21T19:48:32.117 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:31.895576+0000 mon.smithi169 (mon.0) 106 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' 2024-01-21T19:48:32.367 INFO:teuthology.orchestra.run.smithi169.stderr:mgr epoch 13 is available 2024-01-21T19:48:32.368 INFO:teuthology.orchestra.run.smithi169.stderr:Generating a dashboard self-signed certificate... 2024-01-21T19:48:32.368 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-01-21T19:48:32.734 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.734+0000 7f399dfd7700 1 Processor -- start 2024-01-21T19:48:32.735 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.734+0000 7f399dfd7700 1 -- start start 2024-01-21T19:48:32.735 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.734+0000 7f399dfd7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39980fab80 0x7f39980fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:32.735 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.734+0000 7f399dfd7700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39980fb490 con 0x7f39980fab80 2024-01-21T19:48:32.735 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.735+0000 7f39977fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39980fab80 0x7f39980fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:32.735 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.735+0000 7f39977fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39980fab80 0x7f39980fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:58682/0 (socket says 172.21.15.169:58682) 2024-01-21T19:48:32.735 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.735+0000 7f39977fe700 1 -- 172.21.15.169:0/3773792561 learned_addr learned my addr 172.21.15.169:0/3773792561 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:32.736 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.735+0000 7f39977fe700 1 -- 172.21.15.169:0/3773792561 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39980fb5d0 con 0x7f39980fab80 2024-01-21T19:48:32.736 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.735+0000 7f39977fe700 1 --2- 172.21.15.169:0/3773792561 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39980fab80 0x7f39980fafa0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f398000b5f0 tx=0x7f3980006ee0).ready entity=mon.0 client_cookie=eaf041425adba7d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:32.736 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.735+0000 7f39967fc700 1 -- 172.21.15.169:0/3773792561 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f398000d600 con 0x7f39980fab80 2024-01-21T19:48:32.736 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.735+0000 7f39967fc700 1 -- 172.21.15.169:0/3773792561 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f398000dc10 con 0x7f39980fab80 2024-01-21T19:48:32.736 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.735+0000 7f39967fc700 1 -- 172.21.15.169:0/3773792561 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3980011cd0 con 0x7f39980fab80 2024-01-21T19:48:32.736 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.736+0000 7f399dfd7700 1 -- 172.21.15.169:0/3773792561 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39980fab80 msgr2=0x7f39980fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:32.737 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.736+0000 7f399dfd7700 1 --2- 172.21.15.169:0/3773792561 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39980fab80 0x7f39980fafa0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f398000b5f0 tx=0x7f3980006ee0).stop 2024-01-21T19:48:32.737 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.736+0000 7f399dfd7700 1 -- 172.21.15.169:0/3773792561 shutdown_connections 2024-01-21T19:48:32.737 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.736+0000 7f399dfd7700 1 --2- 172.21.15.169:0/3773792561 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39980fab80 0x7f39980fafa0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:32.737 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.736+0000 7f399dfd7700 1 -- 172.21.15.169:0/3773792561 >> 172.21.15.169:0/3773792561 conn(0x7f39980f63c0 msgr2=0x7f39980f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:32.737 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.736+0000 7f399dfd7700 1 -- 172.21.15.169:0/3773792561 shutdown_connections 2024-01-21T19:48:32.737 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.736+0000 7f399dfd7700 1 -- 172.21.15.169:0/3773792561 wait complete. 2024-01-21T19:48:32.737 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.736+0000 7f399dfd7700 1 Processor -- start 2024-01-21T19:48:32.738 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.737+0000 7f399dfd7700 1 -- start start 2024-01-21T19:48:32.738 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.737+0000 7f399dfd7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39980fab80 0x7f3998100ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:32.738 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.737+0000 7f399dfd7700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39980ff320 con 0x7f39980fab80 2024-01-21T19:48:32.738 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.737+0000 7f39977fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39980fab80 0x7f3998100ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:32.738 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.737+0000 7f39977fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39980fab80 0x7f3998100ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:58698/0 (socket says 172.21.15.169:58698) 2024-01-21T19:48:32.738 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.737+0000 7f39977fe700 1 -- 172.21.15.169:0/4046633838 learned_addr learned my addr 172.21.15.169:0/4046633838 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:32.738 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.737+0000 7f39977fe700 1 -- 172.21.15.169:0/4046633838 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398000b200 con 0x7f39980fab80 2024-01-21T19:48:32.738 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.737+0000 7f39977fe700 1 --2- 172.21.15.169:0/4046633838 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39980fab80 0x7f3998100ea0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f398000e040 tx=0x7f3980006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:32.738 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.737+0000 7f398ffff700 1 -- 172.21.15.169:0/4046633838 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f398001b410 con 0x7f39980fab80 2024-01-21T19:48:32.738 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.737+0000 7f399dfd7700 1 -- 172.21.15.169:0/4046633838 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39980ff4c0 con 0x7f39980fab80 2024-01-21T19:48:32.738 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.737+0000 7f398ffff700 1 -- 172.21.15.169:0/4046633838 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f398001ba20 con 0x7f39980fab80 2024-01-21T19:48:32.738 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.737+0000 7f398ffff700 1 -- 172.21.15.169:0/4046633838 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f398001a910 con 0x7f39980fab80 2024-01-21T19:48:32.739 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.737+0000 7f399dfd7700 1 -- 172.21.15.169:0/4046633838 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39980ff8d0 con 0x7f39980fab80 2024-01-21T19:48:32.739 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.738+0000 7f398ffff700 1 -- 172.21.15.169:0/4046633838 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f398000f070 con 0x7f39980fab80 2024-01-21T19:48:32.739 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.738+0000 7f398ffff700 1 --2- 172.21.15.169:0/4046633838 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f397802fd50 0x7f3978032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:32.739 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.738+0000 7f398ffff700 1 -- 172.21.15.169:0/4046633838 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f39800412e0 con 0x7f39980fab80 2024-01-21T19:48:32.739 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.738+0000 7f399dfd7700 1 -- 172.21.15.169:0/4046633838 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3998045ad0 con 0x7f39980fab80 2024-01-21T19:48:32.739 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.738+0000 7f3996ffd700 1 --2- 172.21.15.169:0/4046633838 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f397802fd50 0x7f3978032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:32.739 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.739+0000 7f3996ffd700 1 --2- 172.21.15.169:0/4046633838 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f397802fd50 0x7f3978032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f39880096a0 tx=0x7f3988006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:32.739 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.743+0000 7f398ffff700 1 -- 172.21.15.169:0/4046633838 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3980010be0 con 0x7f39980fab80 2024-01-21T19:48:32.906 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.913+0000 7f399dfd7700 1 -- 172.21.15.169:0/4046633838 --> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f3998104a60 con 0x7f397802fd50 2024-01-21T19:48:32.993 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.996+0000 7f398ffff700 1 -- 172.21.15.169:0/4046633838 <== mgr.14162 v2:172.21.15.169:6800/1381810408 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f3998104a60 con 0x7f397802fd50 2024-01-21T19:48:32.993 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: Self-signed certificate created 2024-01-21T19:48:32.993 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.998+0000 7f399dfd7700 1 -- 172.21.15.169:0/4046633838 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f397802fd50 msgr2=0x7f3978032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:32.993 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.998+0000 7f399dfd7700 1 --2- 172.21.15.169:0/4046633838 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f397802fd50 0x7f3978032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f39880096a0 tx=0x7f3988006b40).stop 2024-01-21T19:48:32.993 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.998+0000 7f399dfd7700 1 -- 172.21.15.169:0/4046633838 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39980fab80 msgr2=0x7f3998100ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:32.993 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.998+0000 7f399dfd7700 1 --2- 172.21.15.169:0/4046633838 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39980fab80 0x7f3998100ea0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f398000e040 tx=0x7f3980006ee0).stop 2024-01-21T19:48:32.993 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.998+0000 7f399dfd7700 1 -- 172.21.15.169:0/4046633838 shutdown_connections 2024-01-21T19:48:32.993 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.998+0000 7f399dfd7700 1 --2- 172.21.15.169:0/4046633838 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f397802fd50 0x7f3978032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:32.993 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.998+0000 7f399dfd7700 1 --2- 172.21.15.169:0/4046633838 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f39980fab80 0x7f3998100ea0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:32.993 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.998+0000 7f399dfd7700 1 -- 172.21.15.169:0/4046633838 >> 172.21.15.169:0/4046633838 conn(0x7f39980f63c0 msgr2=0x7f39980f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:32.993 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.999+0000 7f399dfd7700 1 -- 172.21.15.169:0/4046633838 shutdown_connections 2024-01-21T19:48:32.994 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:32.999+0000 7f399dfd7700 1 -- 172.21.15.169:0/4046633838 wait complete. 2024-01-21T19:48:33.157 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:32 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:31.942336+0000 mgr.smithi169.espgue (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-21T19:48:33.157 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:32 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:31.942949+0000 mon.smithi169 (mon.0) 107 : cluster [DBG] mgrmap e15: smithi169.espgue(active, since 1.06569s) 2024-01-21T19:48:33.157 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:32 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:31.953713+0000 mgr.smithi169.espgue (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-21T19:48:33.157 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:32 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:32.914497+0000 mgr.smithi169.espgue (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:48:33.350 INFO:teuthology.orchestra.run.smithi169.stderr:Creating initial admin user... 2024-01-21T19:48:33.351 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpj3x7xgbx:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-01-21T19:48:33.742 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.745+0000 7f43b9aad700 1 Processor -- start 2024-01-21T19:48:33.742 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.745+0000 7f43b9aad700 1 -- start start 2024-01-21T19:48:33.742 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.745+0000 7f43b9aad700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f43b4070430 0x7f43b40fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:33.743 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.745+0000 7f43b9aad700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b40fedd0 con 0x7f43b4070430 2024-01-21T19:48:33.743 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.745+0000 7f43b37fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f43b4070430 0x7f43b40fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:33.743 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.745+0000 7f43b37fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f43b4070430 0x7f43b40fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:58706/0 (socket says 172.21.15.169:58706) 2024-01-21T19:48:33.743 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.745+0000 7f43b37fe700 1 -- 172.21.15.169:0/611462224 learned_addr learned my addr 172.21.15.169:0/611462224 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:33.743 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.746+0000 7f43b37fe700 1 -- 172.21.15.169:0/611462224 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b40fef10 con 0x7f43b4070430 2024-01-21T19:48:33.743 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.746+0000 7f43b37fe700 1 --2- 172.21.15.169:0/611462224 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f43b4070430 0x7f43b40fe8e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f43a800b5f0 tx=0x7f43a8006ee0).ready entity=mon.0 client_cookie=dad477affaad19d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:33.743 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.746+0000 7f43b27fc700 1 -- 172.21.15.169:0/611462224 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43a800d600 con 0x7f43b4070430 2024-01-21T19:48:33.743 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.746+0000 7f43b27fc700 1 -- 172.21.15.169:0/611462224 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43a800dc10 con 0x7f43b4070430 2024-01-21T19:48:33.743 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.746+0000 7f43b27fc700 1 -- 172.21.15.169:0/611462224 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43a8011cd0 con 0x7f43b4070430 2024-01-21T19:48:33.743 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.747+0000 7f43b9aad700 1 -- 172.21.15.169:0/611462224 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f43b4070430 msgr2=0x7f43b40fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:33.743 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.747+0000 7f43b9aad700 1 --2- 172.21.15.169:0/611462224 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f43b4070430 0x7f43b40fe8e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f43a800b5f0 tx=0x7f43a8006ee0).stop 2024-01-21T19:48:33.743 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.747+0000 7f43b9aad700 1 -- 172.21.15.169:0/611462224 shutdown_connections 2024-01-21T19:48:33.743 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.747+0000 7f43b9aad700 1 --2- 172.21.15.169:0/611462224 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f43b4070430 0x7f43b40fe8e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:33.744 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.747+0000 7f43b9aad700 1 -- 172.21.15.169:0/611462224 >> 172.21.15.169:0/611462224 conn(0x7f43b40f63c0 msgr2=0x7f43b40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:33.744 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.747+0000 7f43b9aad700 1 -- 172.21.15.169:0/611462224 shutdown_connections 2024-01-21T19:48:33.744 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.747+0000 7f43b9aad700 1 -- 172.21.15.169:0/611462224 wait complete. 2024-01-21T19:48:33.744 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.747+0000 7f43b9aad700 1 Processor -- start 2024-01-21T19:48:33.744 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.747+0000 7f43b9aad700 1 -- start start 2024-01-21T19:48:33.744 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.748+0000 7f43b9aad700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f43b4070430 0x7f43b40fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:33.744 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.748+0000 7f43b9aad700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b40fa7e0 con 0x7f43b4070430 2024-01-21T19:48:33.744 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.748+0000 7f43b37fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f43b4070430 0x7f43b40fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:33.744 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.748+0000 7f43b37fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f43b4070430 0x7f43b40fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:58708/0 (socket says 172.21.15.169:58708) 2024-01-21T19:48:33.744 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.748+0000 7f43b37fe700 1 -- 172.21.15.169:0/2094148056 learned_addr learned my addr 172.21.15.169:0/2094148056 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:33.744 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.748+0000 7f43b37fe700 1 -- 172.21.15.169:0/2094148056 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a800b200 con 0x7f43b4070430 2024-01-21T19:48:33.744 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.748+0000 7f43b37fe700 1 --2- 172.21.15.169:0/2094148056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f43b4070430 0x7f43b40fc360 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f43a8006a40 tx=0x7f43a8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:33.744 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.748+0000 7f439bfff700 1 -- 172.21.15.169:0/2094148056 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43a8011c90 con 0x7f43b4070430 2024-01-21T19:48:33.744 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.748+0000 7f43b9aad700 1 -- 172.21.15.169:0/2094148056 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b40fa980 con 0x7f43b4070430 2024-01-21T19:48:33.745 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.748+0000 7f439bfff700 1 -- 172.21.15.169:0/2094148056 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43a801b8c0 con 0x7f43b4070430 2024-01-21T19:48:33.745 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.748+0000 7f439bfff700 1 -- 172.21.15.169:0/2094148056 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43a801a8c0 con 0x7f43b4070430 2024-01-21T19:48:33.745 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.748+0000 7f43b9aad700 1 -- 172.21.15.169:0/2094148056 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b40fad90 con 0x7f43b4070430 2024-01-21T19:48:33.745 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.749+0000 7f43b9aad700 1 -- 172.21.15.169:0/2094148056 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b4045ad0 con 0x7f43b4070430 2024-01-21T19:48:33.746 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.750+0000 7f439bfff700 1 -- 172.21.15.169:0/2094148056 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f43a800f070 con 0x7f43b4070430 2024-01-21T19:48:33.746 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.750+0000 7f439bfff700 1 --2- 172.21.15.169:0/2094148056 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f43a002feb0 0x7f43a0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:33.746 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.750+0000 7f439bfff700 1 -- 172.21.15.169:0/2094148056 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f43a8040cf0 con 0x7f43b4070430 2024-01-21T19:48:33.746 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.750+0000 7f43b2ffd700 1 --2- 172.21.15.169:0/2094148056 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f43a002feb0 0x7f43a0032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:33.746 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.751+0000 7f43b2ffd700 1 --2- 172.21.15.169:0/2094148056 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f43a002feb0 0x7f43a0032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f43a40096a0 tx=0x7f43a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:33.747 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.754+0000 7f439bfff700 1 -- 172.21.15.169:0/2094148056 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f43a80196c0 con 0x7f43b4070430 2024-01-21T19:48:33.912 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:33.918+0000 7f43b9aad700 1 -- 172.21.15.169:0/2094148056 --> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f43b4058f20 con 0x7f43a002feb0 2024-01-21T19:48:34.131 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.137+0000 7f439bfff700 1 -- 172.21.15.169:0/2094148056 <== mgr.14162 v2:172.21.15.169:6800/1381810408 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f43b4058f20 con 0x7f43a002feb0 2024-01-21T19:48:34.131 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$4dKRIhJwFmXdGUts4wJLU.4f72oTnpPQxcUeGmXaoUsrJPcF5LJ.2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1705866514, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-21T19:48:34.133 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.139+0000 7f43b9aad700 1 -- 172.21.15.169:0/2094148056 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f43a002feb0 msgr2=0x7f43a0032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:34.133 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.139+0000 7f43b9aad700 1 --2- 172.21.15.169:0/2094148056 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f43a002feb0 0x7f43a0032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f43a40096a0 tx=0x7f43a4006b40).stop 2024-01-21T19:48:34.133 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.139+0000 7f43b9aad700 1 -- 172.21.15.169:0/2094148056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f43b4070430 msgr2=0x7f43b40fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:34.133 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.139+0000 7f43b9aad700 1 --2- 172.21.15.169:0/2094148056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f43b4070430 0x7f43b40fc360 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f43a8006a40 tx=0x7f43a8006ee0).stop 2024-01-21T19:48:34.133 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.139+0000 7f43b9aad700 1 -- 172.21.15.169:0/2094148056 shutdown_connections 2024-01-21T19:48:34.134 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.139+0000 7f43b9aad700 1 --2- 172.21.15.169:0/2094148056 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f43a002feb0 0x7f43a0032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:34.134 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.139+0000 7f43b9aad700 1 --2- 172.21.15.169:0/2094148056 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f43b4070430 0x7f43b40fc360 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:34.134 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.139+0000 7f43b9aad700 1 -- 172.21.15.169:0/2094148056 >> 172.21.15.169:0/2094148056 conn(0x7f43b40f63c0 msgr2=0x7f43b40f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:34.134 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.139+0000 7f43b9aad700 1 -- 172.21.15.169:0/2094148056 shutdown_connections 2024-01-21T19:48:34.134 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.139+0000 7f43b9aad700 1 -- 172.21.15.169:0/2094148056 wait complete. 2024-01-21T19:48:34.298 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:33 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:32.995177+0000 mon.smithi169 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' 2024-01-21T19:48:34.298 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:33 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:32.997034+0000 mon.smithi169 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' 2024-01-21T19:48:34.298 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:33 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:33.632305+0000 mon.smithi169 (mon.0) 110 : cluster [DBG] mgrmap e16: smithi169.espgue(active, since 2s) 2024-01-21T19:48:34.299 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:33 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:33.920433+0000 mgr.smithi169.espgue (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:48:34.483 INFO:teuthology.orchestra.run.smithi169.stderr:Fetching dashboard port number... 2024-01-21T19:48:34.484 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpxblg0pyr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4hgafovt:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-01-21T19:48:34.838 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.842+0000 7fa8a7557700 1 Processor -- start 2024-01-21T19:48:34.838 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.842+0000 7fa8a7557700 1 -- start start 2024-01-21T19:48:34.838 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.842+0000 7fa8a7557700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8a00fab80 0x7fa8a00fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:34.838 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.842+0000 7fa8a7557700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a00fb490 con 0x7fa8a00fab80 2024-01-21T19:48:34.838 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.842+0000 7fa8a52f3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8a00fab80 0x7fa8a00fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:34.838 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.842+0000 7fa8a52f3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8a00fab80 0x7fa8a00fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:58710/0 (socket says 172.21.15.169:58710) 2024-01-21T19:48:34.838 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.842+0000 7fa8a52f3700 1 -- 172.21.15.169:0/4141391695 learned_addr learned my addr 172.21.15.169:0/4141391695 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:34.838 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.843+0000 7fa8a52f3700 1 -- 172.21.15.169:0/4141391695 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a00fb5d0 con 0x7fa8a00fab80 2024-01-21T19:48:34.839 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.843+0000 7fa8a52f3700 1 --2- 172.21.15.169:0/4141391695 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8a00fab80 0x7fa8a00fafa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fa89000b5f0 tx=0x7fa890006ee0).ready entity=mon.0 client_cookie=a29c126ade8c8645 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:34.839 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.843+0000 7fa88ffff700 1 -- 172.21.15.169:0/4141391695 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa89000d600 con 0x7fa8a00fab80 2024-01-21T19:48:34.839 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.843+0000 7fa88ffff700 1 -- 172.21.15.169:0/4141391695 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa89000dc10 con 0x7fa8a00fab80 2024-01-21T19:48:34.839 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.843+0000 7fa88ffff700 1 -- 172.21.15.169:0/4141391695 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa890011cd0 con 0x7fa8a00fab80 2024-01-21T19:48:34.839 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.843+0000 7fa8a7557700 1 -- 172.21.15.169:0/4141391695 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8a00fab80 msgr2=0x7fa8a00fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:34.839 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.843+0000 7fa8a7557700 1 --2- 172.21.15.169:0/4141391695 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8a00fab80 0x7fa8a00fafa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fa89000b5f0 tx=0x7fa890006ee0).stop 2024-01-21T19:48:34.839 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.844+0000 7fa8a7557700 1 -- 172.21.15.169:0/4141391695 shutdown_connections 2024-01-21T19:48:34.839 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.844+0000 7fa8a7557700 1 --2- 172.21.15.169:0/4141391695 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8a00fab80 0x7fa8a00fafa0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:34.839 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.844+0000 7fa8a7557700 1 -- 172.21.15.169:0/4141391695 >> 172.21.15.169:0/4141391695 conn(0x7fa8a00f63c0 msgr2=0x7fa8a00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:34.839 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.844+0000 7fa8a7557700 1 -- 172.21.15.169:0/4141391695 shutdown_connections 2024-01-21T19:48:34.839 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.844+0000 7fa8a7557700 1 -- 172.21.15.169:0/4141391695 wait complete. 2024-01-21T19:48:34.839 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.844+0000 7fa8a7557700 1 Processor -- start 2024-01-21T19:48:34.839 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.844+0000 7fa8a7557700 1 -- start start 2024-01-21T19:48:34.839 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.844+0000 7fa8a7557700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8a00fab80 0x7fa8a010ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:34.839 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.844+0000 7fa8a7557700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a010d1b0 con 0x7fa8a00fab80 2024-01-21T19:48:34.840 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.844+0000 7fa8a52f3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8a00fab80 0x7fa8a010ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:34.840 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.844+0000 7fa8a52f3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8a00fab80 0x7fa8a010ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:58712/0 (socket says 172.21.15.169:58712) 2024-01-21T19:48:34.841 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.844+0000 7fa8a52f3700 1 -- 172.21.15.169:0/3655510985 learned_addr learned my addr 172.21.15.169:0/3655510985 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:34.841 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.845+0000 7fa8a52f3700 1 -- 172.21.15.169:0/3655510985 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa89000b200 con 0x7fa8a00fab80 2024-01-21T19:48:34.841 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.845+0000 7fa8a52f3700 1 --2- 172.21.15.169:0/3655510985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8a00fab80 0x7fa8a010ccc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fa89000e040 tx=0x7fa890006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:34.841 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.845+0000 7fa88dffb700 1 -- 172.21.15.169:0/3655510985 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa89001b410 con 0x7fa8a00fab80 2024-01-21T19:48:34.841 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.845+0000 7fa88dffb700 1 -- 172.21.15.169:0/3655510985 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa89001ba20 con 0x7fa8a00fab80 2024-01-21T19:48:34.841 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.845+0000 7fa8a7557700 1 -- 172.21.15.169:0/3655510985 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a010d3b0 con 0x7fa8a00fab80 2024-01-21T19:48:34.841 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.845+0000 7fa88dffb700 1 -- 172.21.15.169:0/3655510985 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa89001a910 con 0x7fa8a00fab80 2024-01-21T19:48:34.841 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.845+0000 7fa8a7557700 1 -- 172.21.15.169:0/3655510985 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a010d7c0 con 0x7fa8a00fab80 2024-01-21T19:48:34.841 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.846+0000 7fa88dffb700 1 -- 172.21.15.169:0/3655510985 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fa89000f070 con 0x7fa8a00fab80 2024-01-21T19:48:34.841 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.846+0000 7fa88dffb700 1 --2- 172.21.15.169:0/3655510985 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa88802ff00 0x7fa8880323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:34.841 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.846+0000 7fa88dffb700 1 -- 172.21.15.169:0/3655510985 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8900412e0 con 0x7fa8a00fab80 2024-01-21T19:48:34.841 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.846+0000 7fa8a4af2700 1 --2- 172.21.15.169:0/3655510985 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa88802ff00 0x7fa8880323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:34.842 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.846+0000 7fa8a7557700 1 -- 172.21.15.169:0/3655510985 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a0045ad0 con 0x7fa8a00fab80 2024-01-21T19:48:34.842 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.847+0000 7fa8a4af2700 1 --2- 172.21.15.169:0/3655510985 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa88802ff00 0x7fa8880323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fa89c0096a0 tx=0x7fa89c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:34.843 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:34.851+0000 7fa88dffb700 1 -- 172.21.15.169:0/3655510985 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa890024410 con 0x7fa8a00fab80 2024-01-21T19:48:34.995 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:35.001+0000 7fa8a7557700 1 -- 172.21.15.169:0/3655510985 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fa8a00fc900 con 0x7fa8a00fab80 2024-01-21T19:48:34.995 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:35.002+0000 7fa88dffb700 1 -- 172.21.15.169:0/3655510985 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7fa8a00fc900 con 0x7fa8a00fab80 2024-01-21T19:48:34.995 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 8443 2024-01-21T19:48:34.997 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:35.003+0000 7fa8a7557700 1 -- 172.21.15.169:0/3655510985 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa88802ff00 msgr2=0x7fa8880323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:34.997 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:35.003+0000 7fa8a7557700 1 --2- 172.21.15.169:0/3655510985 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa88802ff00 0x7fa8880323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fa89c0096a0 tx=0x7fa89c006b40).stop 2024-01-21T19:48:34.998 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:35.003+0000 7fa8a7557700 1 -- 172.21.15.169:0/3655510985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8a00fab80 msgr2=0x7fa8a010ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:34.998 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:35.003+0000 7fa8a7557700 1 --2- 172.21.15.169:0/3655510985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8a00fab80 0x7fa8a010ccc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fa89000e040 tx=0x7fa890006ee0).stop 2024-01-21T19:48:34.998 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:35.003+0000 7fa8a7557700 1 -- 172.21.15.169:0/3655510985 shutdown_connections 2024-01-21T19:48:34.998 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:35.003+0000 7fa8a7557700 1 --2- 172.21.15.169:0/3655510985 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa88802ff00 0x7fa8880323c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:34.998 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:35.003+0000 7fa8a7557700 1 --2- 172.21.15.169:0/3655510985 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8a00fab80 0x7fa8a010ccc0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:34.998 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:35.003+0000 7fa8a7557700 1 -- 172.21.15.169:0/3655510985 >> 172.21.15.169:0/3655510985 conn(0x7fa8a00f63c0 msgr2=0x7fa8a0070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:34.998 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:35.004+0000 7fa8a7557700 1 -- 172.21.15.169:0/3655510985 shutdown_connections 2024-01-21T19:48:34.998 INFO:teuthology.orchestra.run.smithi169.stderr:/usr/bin/ceph: 2024-01-21T19:48:35.004+0000 7fa8a7557700 1 -- 172.21.15.169:0/3655510985 wait complete. 2024-01-21T19:48:35.310 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:35 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:34.138627+0000 mon.smithi169 (mon.0) 111 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' 2024-01-21T19:48:35.310 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:35 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:35.002998+0000 mon.smithi169 (mon.0) 112 : audit [DBG] from='client.? 172.21.15.169:0/3655510985' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-21T19:48:35.341 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T19:48:35.349 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: disabled 2024-01-21T19:48:35.350 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T19:48:35.357 INFO:teuthology.orchestra.run.smithi169.stderr:systemctl: inactive 2024-01-21T19:48:35.357 INFO:teuthology.orchestra.run.smithi169.stderr:firewalld.service is not enabled 2024-01-21T19:48:35.357 INFO:teuthology.orchestra.run.smithi169.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-21T19:48:35.360 INFO:teuthology.orchestra.run.smithi169.stderr:Ceph Dashboard is now available at: 2024-01-21T19:48:35.360 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-01-21T19:48:35.361 INFO:teuthology.orchestra.run.smithi169.stderr: URL: https://smithi169.front.sepia.ceph.com:8443/ 2024-01-21T19:48:35.361 INFO:teuthology.orchestra.run.smithi169.stderr: User: admin 2024-01-21T19:48:35.361 INFO:teuthology.orchestra.run.smithi169.stderr: Password: 0asn6bdhyh 2024-01-21T19:48:35.361 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-01-21T19:48:35.361 INFO:teuthology.orchestra.run.smithi169.stderr:You can access the Ceph CLI with: 2024-01-21T19:48:35.361 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-01-21T19:48:35.361 INFO:teuthology.orchestra.run.smithi169.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid df2470f2-b895-11ee-95b0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-21T19:48:35.361 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-01-21T19:48:35.361 INFO:teuthology.orchestra.run.smithi169.stderr:Please consider enabling telemetry to help improve Ceph: 2024-01-21T19:48:35.361 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-01-21T19:48:35.361 INFO:teuthology.orchestra.run.smithi169.stderr: ceph telemetry on 2024-01-21T19:48:35.361 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-01-21T19:48:35.361 INFO:teuthology.orchestra.run.smithi169.stderr:For more information see: 2024-01-21T19:48:35.361 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-01-21T19:48:35.361 INFO:teuthology.orchestra.run.smithi169.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-01-21T19:48:35.361 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-01-21T19:48:35.362 INFO:teuthology.orchestra.run.smithi169.stderr:Bootstrap complete. 2024-01-21T19:48:35.362 INFO:teuthology.orchestra.run.smithi169.stderr:Releasing lock 140394404059624 on /run/cephadm/df2470f2-b895-11ee-95b0-87774f69a715.lock 2024-01-21T19:48:35.362 INFO:teuthology.orchestra.run.smithi169.stderr:Lock 140394404059624 released on /run/cephadm/df2470f2-b895-11ee-95b0-87774f69a715.lock 2024-01-21T19:48:35.394 INFO:tasks.cephadm:Fetching config... 2024-01-21T19:48:35.394 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-01-21T19:48:35.394 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-21T19:48:35.452 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-21T19:48:35.453 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-01-21T19:48:35.453 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-21T19:48:35.511 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-21T19:48:35.512 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-01-21T19:48:35.512 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/mon.smithi169/keyring of=/dev/stdout 2024-01-21T19:48:35.580 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-21T19:48:35.580 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-01-21T19:48:35.580 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-21T19:48:35.637 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-21T19:48:35.638 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCietmYFToaJmPcbdPHcOmy9X42250DP3qHK5+K+bMtLXnyGzcLoanwMtoeJCt0tGfULcTaa0jrynfe9jvQXV79ZOQsacdy6wlsmtCG1TukjkgBYE/DpdZuHDtLgJneMGI7IfNcRIosumvWqAz7ENYIn1s+fARbd1DSoJE7JdAFroqwH7bLatsjd8gs8F/BFzHvwJjzSu0ayEfvI45oZU7PxybCBFPRh8gbVhKmxZn9IAw0GwuBw4aiN72JLymqq5zxWMy6CuMw9g0GrHTbQfoXyw/87oitQr9tk9Auvfm/ohRrhSmUC8oAvdUJ7XIXQxV2qrlAZHXn67weC6yc4GzjHDAl/gOIy0mnULHkEie/+Uj7ZjablRtP9dzkxIPAE2Uq9SKChWAZ7mfHebA7DBD1Kdhja5GaOZ7cAE1WknlNT4SaVu7kfwkq0cju130bsqzxQCGaueWAgfu1aSv9nydEt+6z7NA1Cmi8DzApO3Z6Z++WpQ5bFTkFUaSQZyDEc1k= ceph-df2470f2-b895-11ee-95b0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-21T19:48:35.714 INFO:teuthology.orchestra.run.smithi169.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCietmYFToaJmPcbdPHcOmy9X42250DP3qHK5+K+bMtLXnyGzcLoanwMtoeJCt0tGfULcTaa0jrynfe9jvQXV79ZOQsacdy6wlsmtCG1TukjkgBYE/DpdZuHDtLgJneMGI7IfNcRIosumvWqAz7ENYIn1s+fARbd1DSoJE7JdAFroqwH7bLatsjd8gs8F/BFzHvwJjzSu0ayEfvI45oZU7PxybCBFPRh8gbVhKmxZn9IAw0GwuBw4aiN72JLymqq5zxWMy6CuMw9g0GrHTbQfoXyw/87oitQr9tk9Auvfm/ohRrhSmUC8oAvdUJ7XIXQxV2qrlAZHXn67weC6yc4GzjHDAl/gOIy0mnULHkEie/+Uj7ZjablRtP9dzkxIPAE2Uq9SKChWAZ7mfHebA7DBD1Kdhja5GaOZ7cAE1WknlNT4SaVu7kfwkq0cju130bsqzxQCGaueWAgfu1aSv9nydEt+6z7NA1Cmi8DzApO3Z6Z++WpQ5bFTkFUaSQZyDEc1k= ceph-df2470f2-b895-11ee-95b0-87774f69a715 2024-01-21T19:48:35.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCietmYFToaJmPcbdPHcOmy9X42250DP3qHK5+K+bMtLXnyGzcLoanwMtoeJCt0tGfULcTaa0jrynfe9jvQXV79ZOQsacdy6wlsmtCG1TukjkgBYE/DpdZuHDtLgJneMGI7IfNcRIosumvWqAz7ENYIn1s+fARbd1DSoJE7JdAFroqwH7bLatsjd8gs8F/BFzHvwJjzSu0ayEfvI45oZU7PxybCBFPRh8gbVhKmxZn9IAw0GwuBw4aiN72JLymqq5zxWMy6CuMw9g0GrHTbQfoXyw/87oitQr9tk9Auvfm/ohRrhSmUC8oAvdUJ7XIXQxV2qrlAZHXn67weC6yc4GzjHDAl/gOIy0mnULHkEie/+Uj7ZjablRtP9dzkxIPAE2Uq9SKChWAZ7mfHebA7DBD1Kdhja5GaOZ7cAE1WknlNT4SaVu7kfwkq0cju130bsqzxQCGaueWAgfu1aSv9nydEt+6z7NA1Cmi8DzApO3Z6Z++WpQ5bFTkFUaSQZyDEc1k= ceph-df2470f2-b895-11ee-95b0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-21T19:48:35.769 INFO:teuthology.orchestra.run.smithi173.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCietmYFToaJmPcbdPHcOmy9X42250DP3qHK5+K+bMtLXnyGzcLoanwMtoeJCt0tGfULcTaa0jrynfe9jvQXV79ZOQsacdy6wlsmtCG1TukjkgBYE/DpdZuHDtLgJneMGI7IfNcRIosumvWqAz7ENYIn1s+fARbd1DSoJE7JdAFroqwH7bLatsjd8gs8F/BFzHvwJjzSu0ayEfvI45oZU7PxybCBFPRh8gbVhKmxZn9IAw0GwuBw4aiN72JLymqq5zxWMy6CuMw9g0GrHTbQfoXyw/87oitQr9tk9Auvfm/ohRrhSmUC8oAvdUJ7XIXQxV2qrlAZHXn67weC6yc4GzjHDAl/gOIy0mnULHkEie/+Uj7ZjablRtP9dzkxIPAE2Uq9SKChWAZ7mfHebA7DBD1Kdhja5GaOZ7cAE1WknlNT4SaVu7kfwkq0cju130bsqzxQCGaueWAgfu1aSv9nydEt+6z7NA1Cmi8DzApO3Z6Z++WpQ5bFTkFUaSQZyDEc1k= ceph-df2470f2-b895-11ee-95b0-87774f69a715 2024-01-21T19:48:35.781 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-21T19:48:37.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.223+0000 7f9b7d647700 1 -- 172.21.15.169:0/2299537337 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b780f51a0 msgr2=0x7f9b780f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:37.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.223+0000 7f9b7d647700 1 --2- 172.21.15.169:0/2299537337 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b780f51a0 0x7f9b780f55c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f9b68009fb0 tx=0x7f9b68009b30).stop 2024-01-21T19:48:37.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.223+0000 7f9b7d647700 1 -- 172.21.15.169:0/2299537337 shutdown_connections 2024-01-21T19:48:37.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.223+0000 7f9b7d647700 1 --2- 172.21.15.169:0/2299537337 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b780f51a0 0x7f9b780f55c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:37.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.223+0000 7f9b7d647700 1 -- 172.21.15.169:0/2299537337 >> 172.21.15.169:0/2299537337 conn(0x7f9b780f09a0 msgr2=0x7f9b780f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:37.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.223+0000 7f9b7d647700 1 -- 172.21.15.169:0/2299537337 shutdown_connections 2024-01-21T19:48:37.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.223+0000 7f9b7d647700 1 -- 172.21.15.169:0/2299537337 wait complete. 2024-01-21T19:48:37.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.223+0000 7f9b7d647700 1 Processor -- start 2024-01-21T19:48:37.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.223+0000 7f9b7d647700 1 -- start start 2024-01-21T19:48:37.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.224+0000 7f9b7d647700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b780f51a0 0x7f9b781064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:37.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.224+0000 7f9b7d647700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b78106990 con 0x7f9b780f51a0 2024-01-21T19:48:37.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.224+0000 7f9b77fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b780f51a0 0x7f9b781064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:37.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.224+0000 7f9b77fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b780f51a0 0x7f9b781064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:58740/0 (socket says 172.21.15.169:58740) 2024-01-21T19:48:37.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.224+0000 7f9b77fff700 1 -- 172.21.15.169:0/2387433077 learned_addr learned my addr 172.21.15.169:0/2387433077 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:48:37.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.224+0000 7f9b77fff700 1 -- 172.21.15.169:0/2387433077 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b68005040 con 0x7f9b780f51a0 2024-01-21T19:48:37.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.224+0000 7f9b77fff700 1 --2- 172.21.15.169:0/2387433077 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b780f51a0 0x7f9b781064a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f9b68009830 tx=0x7f9b6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:37.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.224+0000 7f9b74ff9700 1 -- 172.21.15.169:0/2387433077 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b68009320 con 0x7f9b780f51a0 2024-01-21T19:48:37.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.224+0000 7f9b74ff9700 1 -- 172.21.15.169:0/2387433077 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b680136c0 con 0x7f9b780f51a0 2024-01-21T19:48:37.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.224+0000 7f9b7d647700 1 -- 172.21.15.169:0/2387433077 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b78106b90 con 0x7f9b780f51a0 2024-01-21T19:48:37.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.224+0000 7f9b74ff9700 1 -- 172.21.15.169:0/2387433077 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b6801ee90 con 0x7f9b780f51a0 2024-01-21T19:48:37.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.224+0000 7f9b7d647700 1 -- 172.21.15.169:0/2387433077 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b78107020 con 0x7f9b780f51a0 2024-01-21T19:48:37.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.225+0000 7f9b74ff9700 1 -- 172.21.15.169:0/2387433077 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b68014440 con 0x7f9b780f51a0 2024-01-21T19:48:37.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.225+0000 7f9b74ff9700 1 --2- 172.21.15.169:0/2387433077 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9b6002ff10 0x7f9b600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:37.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.225+0000 7f9b7d647700 1 -- 172.21.15.169:0/2387433077 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b78045ad0 con 0x7f9b780f51a0 2024-01-21T19:48:37.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.225+0000 7f9b74ff9700 1 -- 172.21.15.169:0/2387433077 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b68040ee0 con 0x7f9b780f51a0 2024-01-21T19:48:37.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.225+0000 7f9b777fe700 1 --2- 172.21.15.169:0/2387433077 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9b6002ff10 0x7f9b600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:37.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.226+0000 7f9b777fe700 1 --2- 172.21.15.169:0/2387433077 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9b6002ff10 0x7f9b600323d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f9b6c0096a0 tx=0x7f9b6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:37.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.230+0000 7f9b74ff9700 1 -- 172.21.15.169:0/2387433077 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b680273f0 con 0x7f9b780f51a0 2024-01-21T19:48:37.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.389+0000 7f9b7d647700 1 -- 172.21.15.169:0/2387433077 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f9b780fabe0 con 0x7f9b780f51a0 2024-01-21T19:48:37.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.392+0000 7f9b74ff9700 1 -- 172.21.15.169:0/2387433077 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f9b6801c320 con 0x7f9b780f51a0 2024-01-21T19:48:37.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.394+0000 7f9b7d647700 1 -- 172.21.15.169:0/2387433077 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9b6002ff10 msgr2=0x7f9b600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:37.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.394+0000 7f9b7d647700 1 --2- 172.21.15.169:0/2387433077 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9b6002ff10 0x7f9b600323d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f9b6c0096a0 tx=0x7f9b6c006b40).stop 2024-01-21T19:48:37.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.394+0000 7f9b7d647700 1 -- 172.21.15.169:0/2387433077 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b780f51a0 msgr2=0x7f9b781064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:37.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.394+0000 7f9b7d647700 1 --2- 172.21.15.169:0/2387433077 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b780f51a0 0x7f9b781064a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f9b68009830 tx=0x7f9b6800b040).stop 2024-01-21T19:48:37.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.394+0000 7f9b7d647700 1 -- 172.21.15.169:0/2387433077 shutdown_connections 2024-01-21T19:48:37.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.394+0000 7f9b7d647700 1 --2- 172.21.15.169:0/2387433077 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9b6002ff10 0x7f9b600323d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:37.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.394+0000 7f9b7d647700 1 --2- 172.21.15.169:0/2387433077 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b780f51a0 0x7f9b781064a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:37.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.394+0000 7f9b7d647700 1 -- 172.21.15.169:0/2387433077 >> 172.21.15.169:0/2387433077 conn(0x7f9b780f09a0 msgr2=0x7f9b7805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:37.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.394+0000 7f9b7d647700 1 -- 172.21.15.169:0/2387433077 shutdown_connections 2024-01-21T19:48:37.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:48:37.394+0000 7f9b7d647700 1 -- 172.21.15.169:0/2387433077 wait complete. 2024-01-21T19:48:37.773 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi173 2024-01-21T19:48:37.773 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-01-21T19:48:37.773 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.conf 2024-01-21T19:48:37.795 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-01-21T19:48:37.795 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-21T19:48:37.855 INFO:tasks.cephadm:Adding host smithi173 to orchestrator... 2024-01-21T19:48:37.855 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph orch host add smithi173 2024-01-21T19:48:37.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:37 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:36.943535+0000 mon.smithi169 (mon.0) 113 : cluster [DBG] mgrmap e17: smithi169.espgue(active, since 6s) 2024-01-21T19:48:37.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:37 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:37.296053+0000 mon.smithi169 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' 2024-01-21T19:48:38.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:37 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:37.392828+0000 mon.smithi169 (mon.0) 115 : audit [INF] from='client.? 172.21.15.169:0/2387433077' entity='client.admin' 2024-01-21T19:48:38.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:37 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:37.647576+0000 mon.smithi169 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' 2024-01-21T19:48:38.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:37 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cephadm 2024-01-21T19:48:37.653654+0000 mgr.smithi169.espgue (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi169 on smithi169 2024-01-21T19:48:52.416 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:51 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:50.934455+0000 mgr.smithi169.espgue (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:48:54.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:54 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:52.934779+0000 mgr.smithi169.espgue (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:48:56.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:56 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:54.935202+0000 mgr.smithi169.espgue (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:48:58.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.665+0000 7effc752c700 1 -- 172.21.15.173:0/3314918635 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7effc00f6510 msgr2=0x7effc00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:58.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.665+0000 7effc752c700 1 --2- 172.21.15.173:0/3314918635 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7effc00f6510 0x7effc00f6930 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7effb0009fb0 tx=0x7effb0009b30).stop 2024-01-21T19:48:58.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.665+0000 7effc752c700 1 -- 172.21.15.173:0/3314918635 shutdown_connections 2024-01-21T19:48:58.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.665+0000 7effc752c700 1 --2- 172.21.15.173:0/3314918635 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7effc00f6510 0x7effc00f6930 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:58.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.665+0000 7effc752c700 1 -- 172.21.15.173:0/3314918635 >> 172.21.15.173:0/3314918635 conn(0x7effc00f1c70 msgr2=0x7effc00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:58.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.665+0000 7effc752c700 1 -- 172.21.15.173:0/3314918635 shutdown_connections 2024-01-21T19:48:58.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.665+0000 7effc752c700 1 -- 172.21.15.173:0/3314918635 wait complete. 2024-01-21T19:48:58.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.666+0000 7effc752c700 1 Processor -- start 2024-01-21T19:48:58.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.666+0000 7effc752c700 1 -- start start 2024-01-21T19:48:58.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.666+0000 7effc752c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7effc00f6510 0x7effc0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:58.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.666+0000 7effc752c700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc0106c20 con 0x7effc00f6510 2024-01-21T19:48:58.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.666+0000 7effc52c8700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7effc00f6510 0x7effc0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:58.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.666+0000 7effc52c8700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7effc00f6510 0x7effc0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:57800/0 (socket says 172.21.15.173:57800) 2024-01-21T19:48:58.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.666+0000 7effc52c8700 1 -- 172.21.15.173:0/4257770267 learned_addr learned my addr 172.21.15.173:0/4257770267 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:48:58.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.666+0000 7effc52c8700 1 -- 172.21.15.173:0/4257770267 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb0005040 con 0x7effc00f6510 2024-01-21T19:48:58.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.667+0000 7effc52c8700 1 --2- 172.21.15.173:0/4257770267 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7effc00f6510 0x7effc0106730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7effb0013510 tx=0x7effb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:58.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.667+0000 7effb5ffb700 1 -- 172.21.15.173:0/4257770267 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effb0009320 con 0x7effc00f6510 2024-01-21T19:48:58.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.667+0000 7effc752c700 1 -- 172.21.15.173:0/4257770267 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc0106e20 con 0x7effc00f6510 2024-01-21T19:48:58.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.667+0000 7effb5ffb700 1 -- 172.21.15.173:0/4257770267 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7effb0013f00 con 0x7effc00f6510 2024-01-21T19:48:58.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.667+0000 7effb5ffb700 1 -- 172.21.15.173:0/4257770267 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effb0014d90 con 0x7effc00f6510 2024-01-21T19:48:58.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.667+0000 7effc752c700 1 -- 172.21.15.173:0/4257770267 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc01072b0 con 0x7effc00f6510 2024-01-21T19:48:58.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.668+0000 7effb5ffb700 1 -- 172.21.15.173:0/4257770267 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7effb0014420 con 0x7effc00f6510 2024-01-21T19:48:58.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.668+0000 7effc752c700 1 -- 172.21.15.173:0/4257770267 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc0045ad0 con 0x7effc00f6510 2024-01-21T19:48:58.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.668+0000 7effb5ffb700 1 --2- 172.21.15.173:0/4257770267 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7effac02ff60 0x7effac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:48:58.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.668+0000 7effb5ffb700 1 -- 172.21.15.173:0/4257770267 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7effb00410c0 con 0x7effc00f6510 2024-01-21T19:48:58.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.668+0000 7effc4ac7700 1 --2- 172.21.15.173:0/4257770267 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7effac02ff60 0x7effac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:48:58.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.669+0000 7effc4ac7700 1 --2- 172.21.15.173:0/4257770267 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7effac02ff60 0x7effac032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7effbc0096a0 tx=0x7effbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:48:58.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.672+0000 7effb5ffb700 1 -- 172.21.15.173:0/4257770267 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7effb00095f0 con 0x7effc00f6510 2024-01-21T19:48:58.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:58.845+0000 7effc752c700 1 -- 172.21.15.173:0/4257770267 --> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}) v1 -- 0x7effc00f2e70 con 0x7effac02ff60 2024-01-21T19:48:58.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:56.935557+0000 mgr.smithi169.espgue (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:48:59.426 INFO:teuthology.orchestra.run.smithi173.stdout:Added host 'smithi173' 2024-01-21T19:48:59.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:59.433+0000 7effb5ffb700 1 -- 172.21.15.173:0/4257770267 <== mgr.14162 v2:172.21.15.169:6800/1381810408 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7effc00f2e70 con 0x7effac02ff60 2024-01-21T19:48:59.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:59.435+0000 7effc752c700 1 -- 172.21.15.173:0/4257770267 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7effac02ff60 msgr2=0x7effac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:59.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:59.435+0000 7effc752c700 1 --2- 172.21.15.173:0/4257770267 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7effac02ff60 0x7effac032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7effbc0096a0 tx=0x7effbc006b40).stop 2024-01-21T19:48:59.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:59.435+0000 7effc752c700 1 -- 172.21.15.173:0/4257770267 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7effc00f6510 msgr2=0x7effc0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:48:59.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:59.435+0000 7effc752c700 1 --2- 172.21.15.173:0/4257770267 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7effc00f6510 0x7effc0106730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7effb0013510 tx=0x7effb000b040).stop 2024-01-21T19:48:59.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:59.435+0000 7effc752c700 1 -- 172.21.15.173:0/4257770267 shutdown_connections 2024-01-21T19:48:59.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:59.435+0000 7effc752c700 1 --2- 172.21.15.173:0/4257770267 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7effac02ff60 0x7effac032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:59.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:59.435+0000 7effc752c700 1 --2- 172.21.15.173:0/4257770267 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7effc00f6510 0x7effc0106730 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:48:59.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:59.435+0000 7effc752c700 1 -- 172.21.15.173:0/4257770267 >> 172.21.15.173:0/4257770267 conn(0x7effc00f1c70 msgr2=0x7effc00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:48:59.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:59.435+0000 7effc752c700 1 -- 172.21.15.173:0/4257770267 shutdown_connections 2024-01-21T19:48:59.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:48:59.435+0000 7effc752c700 1 -- 172.21.15.173:0/4257770267 wait complete. 2024-01-21T19:48:59.765 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph orch host ls --format=json 2024-01-21T19:48:59.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:59 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:58.846512+0000 mgr.smithi169.espgue (mgr.14162) 10 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:48:59.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:48:59 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:48:59.432718+0000 mon.smithi169 (mon.0) 117 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' 2024-01-21T19:49:00.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:00 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:48:58.935864+0000 mgr.smithi169.espgue (mgr.14162) 11 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:00.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:00 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cephadm 2024-01-21T19:48:59.433266+0000 mgr.smithi169.espgue (mgr.14162) 12 : cephadm [INF] Added host smithi173 2024-01-21T19:49:00.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.949+0000 7f64acad2700 1 -- 172.21.15.173:0/1929710113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64a8060280 msgr2=0x7f64a80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:00.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.949+0000 7f64acad2700 1 --2- 172.21.15.173:0/1929710113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64a8060280 0x7f64a80f90f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f649c009fb0 tx=0x7f649c009b30).stop 2024-01-21T19:49:00.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.949+0000 7f64acad2700 1 -- 172.21.15.173:0/1929710113 shutdown_connections 2024-01-21T19:49:00.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.949+0000 7f64acad2700 1 --2- 172.21.15.173:0/1929710113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64a8060280 0x7f64a80f90f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:00.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.949+0000 7f64acad2700 1 -- 172.21.15.173:0/1929710113 >> 172.21.15.173:0/1929710113 conn(0x7f64a80f0b70 msgr2=0x7f64a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:00.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.949+0000 7f64acad2700 1 -- 172.21.15.173:0/1929710113 shutdown_connections 2024-01-21T19:49:00.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.949+0000 7f64acad2700 1 -- 172.21.15.173:0/1929710113 wait complete. 2024-01-21T19:49:00.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.950+0000 7f64acad2700 1 Processor -- start 2024-01-21T19:49:00.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.950+0000 7f64acad2700 1 -- start start 2024-01-21T19:49:00.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.950+0000 7f64acad2700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64a8060280 0x7f64a81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:00.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.950+0000 7f64acad2700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a8106bd0 con 0x7f64a8060280 2024-01-21T19:49:00.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.950+0000 7f64a659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64a8060280 0x7f64a81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:00.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.951+0000 7f64a659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64a8060280 0x7f64a81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:57814/0 (socket says 172.21.15.173:57814) 2024-01-21T19:49:00.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.951+0000 7f64a659c700 1 -- 172.21.15.173:0/354696295 learned_addr learned my addr 172.21.15.173:0/354696295 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:49:00.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.951+0000 7f64a659c700 1 -- 172.21.15.173:0/354696295 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649c005040 con 0x7f64a8060280 2024-01-21T19:49:00.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.951+0000 7f64a659c700 1 --2- 172.21.15.173:0/354696295 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64a8060280 0x7f64a81066e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f649c013510 tx=0x7f649c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:00.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.951+0000 7f6496ffd700 1 -- 172.21.15.173:0/354696295 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f649c009320 con 0x7f64a8060280 2024-01-21T19:49:00.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.951+0000 7f6496ffd700 1 -- 172.21.15.173:0/354696295 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f649c013f00 con 0x7f64a8060280 2024-01-21T19:49:00.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.951+0000 7f64acad2700 1 -- 172.21.15.173:0/354696295 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a8106dd0 con 0x7f64a8060280 2024-01-21T19:49:00.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.951+0000 7f6496ffd700 1 -- 172.21.15.173:0/354696295 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f649c014dc0 con 0x7f64a8060280 2024-01-21T19:49:00.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.951+0000 7f64acad2700 1 -- 172.21.15.173:0/354696295 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a8107260 con 0x7f64a8060280 2024-01-21T19:49:00.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.952+0000 7f6496ffd700 1 -- 172.21.15.173:0/354696295 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f649c014420 con 0x7f64a8060280 2024-01-21T19:49:00.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.952+0000 7f6496ffd700 1 --2- 172.21.15.173:0/354696295 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f649002ff60 0x7f6490032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:00.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.952+0000 7f6496ffd700 1 -- 172.21.15.173:0/354696295 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f649c0410f0 con 0x7f64a8060280 2024-01-21T19:49:00.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.952+0000 7f64acad2700 1 -- 172.21.15.173:0/354696295 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a8045ad0 con 0x7f64a8060280 2024-01-21T19:49:00.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.952+0000 7f64a5d9b700 1 --2- 172.21.15.173:0/354696295 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f649002ff60 0x7f6490032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:00.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.953+0000 7f64a5d9b700 1 --2- 172.21.15.173:0/354696295 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f649002ff60 0x7f6490032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f64980096a0 tx=0x7f6498006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:00.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:00.957+0000 7f6496ffd700 1 -- 172.21.15.173:0/354696295 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f649c0095f0 con 0x7f64a8060280 2024-01-21T19:49:01.118 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:01.126+0000 7f64acad2700 1 -- 172.21.15.173:0/354696295 --> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f64a80f1d70 con 0x7f649002ff60 2024-01-21T19:49:01.120 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:01.127+0000 7f6496ffd700 1 -- 172.21.15.173:0/354696295 <== mgr.14162 v2:172.21.15.169:6800/1381810408 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+157 (secure 0 0 0) 0x7f64a80f1d70 con 0x7f649002ff60 2024-01-21T19:49:01.120 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:49:01.120 INFO:teuthology.orchestra.run.smithi173.stdout:[{"addr": "172.21.15.169", "hostname": "smithi169", "labels": [], "status": ""}, {"addr": "smithi173", "hostname": "smithi173", "labels": [], "status": ""}] 2024-01-21T19:49:01.122 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:01.129+0000 7f64acad2700 1 -- 172.21.15.173:0/354696295 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f649002ff60 msgr2=0x7f6490032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:01.122 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:01.129+0000 7f64acad2700 1 --2- 172.21.15.173:0/354696295 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f649002ff60 0x7f6490032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f64980096a0 tx=0x7f6498006b40).stop 2024-01-21T19:49:01.122 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:01.129+0000 7f64acad2700 1 -- 172.21.15.173:0/354696295 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64a8060280 msgr2=0x7f64a81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:01.122 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:01.129+0000 7f64acad2700 1 --2- 172.21.15.173:0/354696295 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64a8060280 0x7f64a81066e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f649c013510 tx=0x7f649c00b040).stop 2024-01-21T19:49:01.122 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:01.129+0000 7f64acad2700 1 -- 172.21.15.173:0/354696295 shutdown_connections 2024-01-21T19:49:01.122 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:01.129+0000 7f64acad2700 1 --2- 172.21.15.173:0/354696295 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f649002ff60 0x7f6490032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:01.122 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:01.129+0000 7f64acad2700 1 --2- 172.21.15.173:0/354696295 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64a8060280 0x7f64a81066e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:01.122 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:01.129+0000 7f64acad2700 1 -- 172.21.15.173:0/354696295 >> 172.21.15.173:0/354696295 conn(0x7f64a80f0b70 msgr2=0x7f64a80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:01.122 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:01.129+0000 7f64acad2700 1 -- 172.21.15.173:0/354696295 shutdown_connections 2024-01-21T19:49:01.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:01.129+0000 7f64acad2700 1 -- 172.21.15.173:0/354696295 wait complete. 2024-01-21T19:49:01.487 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-21T19:49:01.487 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph osd crush tunables default 2024-01-21T19:49:02.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.669+0000 7f4a60269700 1 -- 172.21.15.169:0/4133389173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a580f5410 msgr2=0x7f4a580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:02.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.669+0000 7f4a60269700 1 --2- 172.21.15.169:0/4133389173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a580f5410 0x7f4a580f5830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f4a48009fb0 tx=0x7f4a48009b30).stop 2024-01-21T19:49:02.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.669+0000 7f4a60269700 1 -- 172.21.15.169:0/4133389173 shutdown_connections 2024-01-21T19:49:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.669+0000 7f4a60269700 1 --2- 172.21.15.169:0/4133389173 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a580f5410 0x7f4a580f5830 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.669+0000 7f4a60269700 1 -- 172.21.15.169:0/4133389173 >> 172.21.15.169:0/4133389173 conn(0x7f4a580f0b70 msgr2=0x7f4a580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.669+0000 7f4a60269700 1 -- 172.21.15.169:0/4133389173 shutdown_connections 2024-01-21T19:49:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.669+0000 7f4a60269700 1 -- 172.21.15.169:0/4133389173 wait complete. 2024-01-21T19:49:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.670+0000 7f4a60269700 1 Processor -- start 2024-01-21T19:49:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.670+0000 7f4a60269700 1 -- start start 2024-01-21T19:49:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.670+0000 7f4a60269700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a580f5410 0x7f4a58105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.670+0000 7f4a60269700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a58105b20 con 0x7f4a580f5410 2024-01-21T19:49:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.670+0000 7f4a5e005700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a580f5410 0x7f4a58105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.670+0000 7f4a5e005700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a580f5410 0x7f4a58105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:41652/0 (socket says 172.21.15.169:41652) 2024-01-21T19:49:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.670+0000 7f4a5e005700 1 -- 172.21.15.169:0/1144371876 learned_addr learned my addr 172.21.15.169:0/1144371876 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-21T19:49:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.670+0000 7f4a5e005700 1 -- 172.21.15.169:0/1144371876 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a48005040 con 0x7f4a580f5410 2024-01-21T19:49:02.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.670+0000 7f4a5e005700 1 --2- 172.21.15.169:0/1144371876 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a580f5410 0x7f4a58105630 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f4a48014000 tx=0x7f4a4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:02.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.670+0000 7f4a4effd700 1 -- 172.21.15.169:0/1144371876 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a480095f0 con 0x7f4a580f5410 2024-01-21T19:49:02.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.670+0000 7f4a4effd700 1 -- 172.21.15.169:0/1144371876 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a48007e70 con 0x7f4a580f5410 2024-01-21T19:49:02.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.670+0000 7f4a60269700 1 -- 172.21.15.169:0/1144371876 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a58105d20 con 0x7f4a580f5410 2024-01-21T19:49:02.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.670+0000 7f4a4effd700 1 -- 172.21.15.169:0/1144371876 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a48014ea0 con 0x7f4a580f5410 2024-01-21T19:49:02.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.670+0000 7f4a60269700 1 -- 172.21.15.169:0/1144371876 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a581061b0 con 0x7f4a580f5410 2024-01-21T19:49:02.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.671+0000 7f4a4effd700 1 -- 172.21.15.169:0/1144371876 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a48027070 con 0x7f4a580f5410 2024-01-21T19:49:02.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.671+0000 7f4a4effd700 1 --2- 172.21.15.169:0/1144371876 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4a4402ffb0 0x7f4a44032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:02.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.671+0000 7f4a60269700 1 -- 172.21.15.169:0/1144371876 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a58045ad0 con 0x7f4a580f5410 2024-01-21T19:49:02.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.671+0000 7f4a4effd700 1 -- 172.21.15.169:0/1144371876 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a480426d0 con 0x7f4a580f5410 2024-01-21T19:49:02.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.672+0000 7f4a5d804700 1 --2- 172.21.15.169:0/1144371876 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4a4402ffb0 0x7f4a44032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:02.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.672+0000 7f4a5d804700 1 --2- 172.21.15.169:0/1144371876 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4a4402ffb0 0x7f4a44032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f4a540096a0 tx=0x7f4a54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:02.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.676+0000 7f4a4effd700 1 -- 172.21.15.169:0/1144371876 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a48027710 con 0x7f4a580f5410 2024-01-21T19:49:02.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:02.831+0000 7f4a60269700 1 -- 172.21.15.169:0/1144371876 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f4a580faf50 con 0x7f4a580f5410 2024-01-21T19:49:02.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:02 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:00.936123+0000 mgr.smithi169.espgue (mgr.14162) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:02.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:02 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:01.127558+0000 mgr.smithi169.espgue (mgr.14162) 14 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-21T19:49:03.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:03.655+0000 7f4a4effd700 1 -- 172.21.15.169:0/1144371876 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f4a4801e320 con 0x7f4a580f5410 2024-01-21T19:49:03.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:03.657+0000 7f4a60269700 1 -- 172.21.15.169:0/1144371876 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4a4402ffb0 msgr2=0x7f4a44032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:03.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:03.657+0000 7f4a60269700 1 --2- 172.21.15.169:0/1144371876 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4a4402ffb0 0x7f4a44032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f4a540096a0 tx=0x7f4a54006b40).stop 2024-01-21T19:49:03.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:03.657+0000 7f4a60269700 1 -- 172.21.15.169:0/1144371876 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a580f5410 msgr2=0x7f4a58105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:03.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:03.657+0000 7f4a60269700 1 --2- 172.21.15.169:0/1144371876 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a580f5410 0x7f4a58105630 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f4a48014000 tx=0x7f4a4800b040).stop 2024-01-21T19:49:03.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:03.658+0000 7f4a60269700 1 -- 172.21.15.169:0/1144371876 shutdown_connections 2024-01-21T19:49:03.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:03.658+0000 7f4a60269700 1 --2- 172.21.15.169:0/1144371876 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4a4402ffb0 0x7f4a44032470 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:03.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:03.658+0000 7f4a60269700 1 --2- 172.21.15.169:0/1144371876 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a580f5410 0x7f4a58105630 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:03.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:03.658+0000 7f4a60269700 1 -- 172.21.15.169:0/1144371876 >> 172.21.15.169:0/1144371876 conn(0x7f4a580f0b70 msgr2=0x7f4a580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:03.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:03.658+0000 7f4a60269700 1 -- 172.21.15.169:0/1144371876 shutdown_connections 2024-01-21T19:49:03.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-21T19:49:03.658+0000 7f4a60269700 1 -- 172.21.15.169:0/1144371876 wait complete. 2024-01-21T19:49:03.652 INFO:teuthology.orchestra.run.smithi169.stderr:adjusted tunables profile to default 2024-01-21T19:49:03.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:03 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:02.832386+0000 mon.smithi169 (mon.0) 118 : audit [INF] from='client.? 172.21.15.169:0/1144371876' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-21T19:49:04.028 INFO:tasks.cephadm:Adding mon.smithi169 on smithi169 2024-01-21T19:49:04.028 INFO:tasks.cephadm:Adding mon.smithi173 on smithi173 2024-01-21T19:49:04.028 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph orch apply mon '2;smithi169:172.21.15.169=smithi169;smithi173:172.21.15.173=smithi173' 2024-01-21T19:49:04.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:04 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:02.936346+0000 mgr.smithi169.espgue (mgr.14162) 15 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:04.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:04 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:03.656196+0000 mon.smithi169 (mon.0) 119 : audit [INF] from='client.? 172.21.15.169:0/1144371876' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-21T19:49:04.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:04 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:03.656302+0000 mon.smithi169 (mon.0) 120 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-21T19:49:05.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.203+0000 7f1e2ebe5700 1 -- 172.21.15.173:0/1593045864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e280f5410 msgr2=0x7f1e280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:05.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.203+0000 7f1e2ebe5700 1 --2- 172.21.15.173:0/1593045864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e280f5410 0x7f1e280f5830 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f1e1c009fb0 tx=0x7f1e1c009b30).stop 2024-01-21T19:49:05.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.203+0000 7f1e2ebe5700 1 -- 172.21.15.173:0/1593045864 shutdown_connections 2024-01-21T19:49:05.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.203+0000 7f1e2ebe5700 1 --2- 172.21.15.173:0/1593045864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e280f5410 0x7f1e280f5830 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:05.197 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.203+0000 7f1e2ebe5700 1 -- 172.21.15.173:0/1593045864 >> 172.21.15.173:0/1593045864 conn(0x7f1e280f0b70 msgr2=0x7f1e280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:05.197 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.203+0000 7f1e2ebe5700 1 -- 172.21.15.173:0/1593045864 shutdown_connections 2024-01-21T19:49:05.197 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.204+0000 7f1e2ebe5700 1 -- 172.21.15.173:0/1593045864 wait complete. 2024-01-21T19:49:05.198 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.204+0000 7f1e2ebe5700 1 Processor -- start 2024-01-21T19:49:05.198 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.204+0000 7f1e2ebe5700 1 -- start start 2024-01-21T19:49:05.198 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.205+0000 7f1e2ebe5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e280f5410 0x7f1e28109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:05.198 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.205+0000 7f1e2ebe5700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e28060240 con 0x7f1e280f5410 2024-01-21T19:49:05.198 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.205+0000 7f1e2c981700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e280f5410 0x7f1e28109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:05.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.205+0000 7f1e2c981700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e280f5410 0x7f1e28109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:45694/0 (socket says 172.21.15.173:45694) 2024-01-21T19:49:05.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.205+0000 7f1e2c981700 1 -- 172.21.15.173:0/1156929307 learned_addr learned my addr 172.21.15.173:0/1156929307 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:49:05.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.205+0000 7f1e2c981700 1 -- 172.21.15.173:0/1156929307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1c005040 con 0x7f1e280f5410 2024-01-21T19:49:05.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.205+0000 7f1e2c981700 1 --2- 172.21.15.173:0/1156929307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e280f5410 0x7f1e28109430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f1e1c013510 tx=0x7f1e1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:05.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.205+0000 7f1e257fa700 1 -- 172.21.15.173:0/1156929307 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e1c009320 con 0x7f1e280f5410 2024-01-21T19:49:05.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.206+0000 7f1e2ebe5700 1 -- 172.21.15.173:0/1156929307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e28060440 con 0x7f1e280f5410 2024-01-21T19:49:05.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.206+0000 7f1e257fa700 1 -- 172.21.15.173:0/1156929307 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e1c013f00 con 0x7f1e280f5410 2024-01-21T19:49:05.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.206+0000 7f1e257fa700 1 -- 172.21.15.173:0/1156929307 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e1c014d90 con 0x7f1e280f5410 2024-01-21T19:49:05.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.206+0000 7f1e2ebe5700 1 -- 172.21.15.173:0/1156929307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e28109ae0 con 0x7f1e280f5410 2024-01-21T19:49:05.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.206+0000 7f1e257fa700 1 -- 172.21.15.173:0/1156929307 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e1c0263a0 con 0x7f1e280f5410 2024-01-21T19:49:05.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.206+0000 7f1e257fa700 1 --2- 172.21.15.173:0/1156929307 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1e1802ff60 0x7f1e18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:05.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.206+0000 7f1e2ebe5700 1 -- 172.21.15.173:0/1156929307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e28045ad0 con 0x7f1e280f5410 2024-01-21T19:49:05.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.206+0000 7f1e257fa700 1 -- 172.21.15.173:0/1156929307 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e1c041390 con 0x7f1e280f5410 2024-01-21T19:49:05.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.207+0000 7f1e27fff700 1 --2- 172.21.15.173:0/1156929307 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1e1802ff60 0x7f1e18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:05.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.207+0000 7f1e27fff700 1 --2- 172.21.15.173:0/1156929307 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1e1802ff60 0x7f1e18032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f1e140096a0 tx=0x7f1e14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:05.203 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.211+0000 7f1e257fa700 1 -- 172.21.15.173:0/1156929307 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e1c012330 con 0x7f1e280f5410 2024-01-21T19:49:05.381 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.388+0000 7f1e2ebe5700 1 -- 172.21.15.173:0/1156929307 --> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi169:172.21.15.169=smithi169;smithi173:172.21.15.173=smithi173", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e280f1e20 con 0x7f1e1802ff60 2024-01-21T19:49:05.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.394+0000 7f1e257fa700 1 -- 172.21.15.173:0/1156929307 <== mgr.14162 v2:172.21.15.169:6800/1381810408 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1e280f1e20 con 0x7f1e1802ff60 2024-01-21T19:49:05.387 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mon update... 2024-01-21T19:49:05.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.396+0000 7f1e2ebe5700 1 -- 172.21.15.173:0/1156929307 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1e1802ff60 msgr2=0x7f1e18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:05.389 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.396+0000 7f1e2ebe5700 1 --2- 172.21.15.173:0/1156929307 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1e1802ff60 0x7f1e18032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f1e140096a0 tx=0x7f1e14006b40).stop 2024-01-21T19:49:05.389 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.396+0000 7f1e2ebe5700 1 -- 172.21.15.173:0/1156929307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e280f5410 msgr2=0x7f1e28109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:05.389 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.396+0000 7f1e2ebe5700 1 --2- 172.21.15.173:0/1156929307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e280f5410 0x7f1e28109430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f1e1c013510 tx=0x7f1e1c00b040).stop 2024-01-21T19:49:05.389 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.396+0000 7f1e2ebe5700 1 -- 172.21.15.173:0/1156929307 shutdown_connections 2024-01-21T19:49:05.389 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.396+0000 7f1e2ebe5700 1 --2- 172.21.15.173:0/1156929307 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1e1802ff60 0x7f1e18032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:05.389 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.396+0000 7f1e2ebe5700 1 --2- 172.21.15.173:0/1156929307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e280f5410 0x7f1e28109430 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:05.389 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.396+0000 7f1e2ebe5700 1 -- 172.21.15.173:0/1156929307 >> 172.21.15.173:0/1156929307 conn(0x7f1e280f0b70 msgr2=0x7f1e280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:05.389 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.396+0000 7f1e2ebe5700 1 -- 172.21.15.173:0/1156929307 shutdown_connections 2024-01-21T19:49:05.389 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:05.396+0000 7f1e2ebe5700 1 -- 172.21.15.173:0/1156929307 wait complete. 2024-01-21T19:49:05.773 DEBUG:teuthology.orchestra.run.smithi173:mon.smithi173> sudo journalctl -f -n 0 -u ceph-df2470f2-b895-11ee-95b0-87774f69a715@mon.smithi173.service 2024-01-21T19:49:05.775 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:49:05.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:49:05.804 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:-- Logs begin at Sun 2024-01-21 19:37:14 UTC. -- 2024-01-21T19:49:06.665 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:06 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:04.936677+0000 mgr.smithi169.espgue (mgr.14162) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:06.666 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:06 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:05.390223+0000 mgr.smithi169.espgue (mgr.14162) 17 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi169:172.21.15.169=smithi169;smithi173:172.21.15.173=smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:49:06.666 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:06 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cephadm 2024-01-21T19:49:05.391530+0000 mgr.smithi169.espgue (mgr.14162) 18 : cephadm [INF] Saving service mon spec with placement smithi169:172.21.15.169=smithi169;smithi173:172.21.15.173=smithi173;count:2 2024-01-21T19:49:06.666 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:06 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:05.394318+0000 mon.smithi169 (mon.0) 121 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' 2024-01-21T19:49:07.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.038+0000 7f0c5e426700 1 -- 172.21.15.173:0/240229303 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c580f62a0 msgr2=0x7f0c580f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:07.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.038+0000 7f0c5e426700 1 --2- 172.21.15.173:0/240229303 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c580f62a0 0x7f0c580f66c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f0c48009fb0 tx=0x7f0c48009b30).stop 2024-01-21T19:49:07.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.039+0000 7f0c5e426700 1 -- 172.21.15.173:0/240229303 shutdown_connections 2024-01-21T19:49:07.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.039+0000 7f0c5e426700 1 --2- 172.21.15.173:0/240229303 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c580f62a0 0x7f0c580f66c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:07.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.039+0000 7f0c5e426700 1 -- 172.21.15.173:0/240229303 >> 172.21.15.173:0/240229303 conn(0x7f0c580f1aa0 msgr2=0x7f0c580f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:07.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.039+0000 7f0c5e426700 1 -- 172.21.15.173:0/240229303 shutdown_connections 2024-01-21T19:49:07.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.039+0000 7f0c5e426700 1 -- 172.21.15.173:0/240229303 wait complete. 2024-01-21T19:49:07.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.039+0000 7f0c5e426700 1 Processor -- start 2024-01-21T19:49:07.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.039+0000 7f0c5e426700 1 -- start start 2024-01-21T19:49:07.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.039+0000 7f0c5e426700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c580f62a0 0x7f0c581064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:07.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.039+0000 7f0c5e426700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c581069a0 con 0x7f0c580f62a0 2024-01-21T19:49:07.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.040+0000 7f0c5d424700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c580f62a0 0x7f0c581064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:07.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.040+0000 7f0c5d424700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c580f62a0 0x7f0c581064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:45704/0 (socket says 172.21.15.173:45704) 2024-01-21T19:49:07.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.040+0000 7f0c5d424700 1 -- 172.21.15.173:0/2825076496 learned_addr learned my addr 172.21.15.173:0/2825076496 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:49:07.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.040+0000 7f0c5d424700 1 -- 172.21.15.173:0/2825076496 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c48005040 con 0x7f0c580f62a0 2024-01-21T19:49:07.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.040+0000 7f0c5d424700 1 --2- 172.21.15.173:0/2825076496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c580f62a0 0x7f0c581064b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f0c48013510 tx=0x7f0c4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:07.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.040+0000 7f0c4dffb700 1 -- 172.21.15.173:0/2825076496 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c48009320 con 0x7f0c580f62a0 2024-01-21T19:49:07.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.040+0000 7f0c4dffb700 1 -- 172.21.15.173:0/2825076496 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c48013f00 con 0x7f0c580f62a0 2024-01-21T19:49:07.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.040+0000 7f0c5e426700 1 -- 172.21.15.173:0/2825076496 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c58106ba0 con 0x7f0c580f62a0 2024-01-21T19:49:07.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.040+0000 7f0c4dffb700 1 -- 172.21.15.173:0/2825076496 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c48014dc0 con 0x7f0c580f62a0 2024-01-21T19:49:07.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.040+0000 7f0c5e426700 1 -- 172.21.15.173:0/2825076496 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c58107030 con 0x7f0c580f62a0 2024-01-21T19:49:07.035 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.041+0000 7f0c4dffb700 1 -- 172.21.15.173:0/2825076496 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c480095f0 con 0x7f0c580f62a0 2024-01-21T19:49:07.035 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.041+0000 7f0c5e426700 1 -- 172.21.15.173:0/2825076496 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c58045ad0 con 0x7f0c580f62a0 2024-01-21T19:49:07.035 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.041+0000 7f0c4dffb700 1 --2- 172.21.15.173:0/2825076496 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0c4402ff60 0x7f0c44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:07.035 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.041+0000 7f0c4dffb700 1 -- 172.21.15.173:0/2825076496 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c48041020 con 0x7f0c580f62a0 2024-01-21T19:49:07.035 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.042+0000 7f0c5cc23700 1 --2- 172.21.15.173:0/2825076496 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0c4402ff60 0x7f0c44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:07.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.042+0000 7f0c5cc23700 1 --2- 172.21.15.173:0/2825076496 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0c4402ff60 0x7f0c44032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f0c540096a0 tx=0x7f0c54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:07.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.046+0000 7f0c4dffb700 1 -- 172.21.15.173:0/2825076496 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c48014420 con 0x7f0c580f62a0 2024-01-21T19:49:07.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.252+0000 7f0c5e426700 1 -- 172.21.15.173:0/2825076496 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c58107e80 con 0x7f0c580f62a0 2024-01-21T19:49:07.245 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.252+0000 7f0c4dffb700 1 -- 172.21.15.173:0/2825076496 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0c4801c920 con 0x7f0c580f62a0 2024-01-21T19:49:07.245 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:49:07.245 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:49:07.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.254+0000 7f0c5e426700 1 -- 172.21.15.173:0/2825076496 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0c4402ff60 msgr2=0x7f0c44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:07.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.254+0000 7f0c5e426700 1 --2- 172.21.15.173:0/2825076496 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0c4402ff60 0x7f0c44032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f0c540096a0 tx=0x7f0c54006b40).stop 2024-01-21T19:49:07.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.254+0000 7f0c5e426700 1 -- 172.21.15.173:0/2825076496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c580f62a0 msgr2=0x7f0c581064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:07.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.254+0000 7f0c5e426700 1 --2- 172.21.15.173:0/2825076496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c580f62a0 0x7f0c581064b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f0c48013510 tx=0x7f0c4800b040).stop 2024-01-21T19:49:07.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.254+0000 7f0c5e426700 1 -- 172.21.15.173:0/2825076496 shutdown_connections 2024-01-21T19:49:07.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.254+0000 7f0c5e426700 1 --2- 172.21.15.173:0/2825076496 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0c4402ff60 0x7f0c44032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:07.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.254+0000 7f0c5e426700 1 --2- 172.21.15.173:0/2825076496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c580f62a0 0x7f0c581064b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:07.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.254+0000 7f0c5e426700 1 -- 172.21.15.173:0/2825076496 >> 172.21.15.173:0/2825076496 conn(0x7f0c580f1aa0 msgr2=0x7f0c580f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:07.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.254+0000 7f0c5e426700 1 -- 172.21.15.173:0/2825076496 shutdown_connections 2024-01-21T19:49:07.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:07.254+0000 7f0c5e426700 1 -- 172.21.15.173:0/2825076496 wait complete. 2024-01-21T19:49:07.248 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:49:07.665 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:07 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:07.252969+0000 mon.smithi169 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.173:0/2825076496' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:49:08.647 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:49:08.647 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:49:08.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:08 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:06.937043+0000 mgr.smithi169.espgue (mgr.14162) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:10.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.118+0000 7fe201169700 1 -- 172.21.15.173:0/732453308 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1fc0f5410 msgr2=0x7fe1fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:10.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.118+0000 7fe201169700 1 --2- 172.21.15.173:0/732453308 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1fc0f5410 0x7fe1fc0f5830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fe1e4009fb0 tx=0x7fe1e4009b30).stop 2024-01-21T19:49:10.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.118+0000 7fe201169700 1 -- 172.21.15.173:0/732453308 shutdown_connections 2024-01-21T19:49:10.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.118+0000 7fe201169700 1 --2- 172.21.15.173:0/732453308 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1fc0f5410 0x7fe1fc0f5830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:10.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.118+0000 7fe201169700 1 -- 172.21.15.173:0/732453308 >> 172.21.15.173:0/732453308 conn(0x7fe1fc0f0b70 msgr2=0x7fe1fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:10.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.118+0000 7fe201169700 1 -- 172.21.15.173:0/732453308 shutdown_connections 2024-01-21T19:49:10.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.118+0000 7fe201169700 1 -- 172.21.15.173:0/732453308 wait complete. 2024-01-21T19:49:10.113 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.119+0000 7fe201169700 1 Processor -- start 2024-01-21T19:49:10.113 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.119+0000 7fe201169700 1 -- start start 2024-01-21T19:49:10.114 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.119+0000 7fe201169700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1fc0f5410 0x7fe1fc109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:10.114 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.119+0000 7fe201169700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc060220 con 0x7fe1fc0f5410 2024-01-21T19:49:10.114 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.119+0000 7fe1fad9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1fc0f5410 0x7fe1fc109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:10.114 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.119+0000 7fe1fad9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1fc0f5410 0x7fe1fc109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:45724/0 (socket says 172.21.15.173:45724) 2024-01-21T19:49:10.114 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.119+0000 7fe1fad9d700 1 -- 172.21.15.173:0/441701735 learned_addr learned my addr 172.21.15.173:0/441701735 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:49:10.114 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.120+0000 7fe1fad9d700 1 -- 172.21.15.173:0/441701735 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e4005040 con 0x7fe1fc0f5410 2024-01-21T19:49:10.114 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.120+0000 7fe1fad9d700 1 --2- 172.21.15.173:0/441701735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1fc0f5410 0x7fe1fc109450 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fe1e4014000 tx=0x7fe1e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:10.114 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.120+0000 7fe1f37fe700 1 -- 172.21.15.173:0/441701735 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1e40095f0 con 0x7fe1fc0f5410 2024-01-21T19:49:10.114 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.120+0000 7fe201169700 1 -- 172.21.15.173:0/441701735 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1fc060420 con 0x7fe1fc0f5410 2024-01-21T19:49:10.114 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.120+0000 7fe1f37fe700 1 -- 172.21.15.173:0/441701735 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe1e4007e70 con 0x7fe1fc0f5410 2024-01-21T19:49:10.114 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.120+0000 7fe1f37fe700 1 -- 172.21.15.173:0/441701735 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1e4014ea0 con 0x7fe1fc0f5410 2024-01-21T19:49:10.114 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.120+0000 7fe201169700 1 -- 172.21.15.173:0/441701735 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1fc109b00 con 0x7fe1fc0f5410 2024-01-21T19:49:10.114 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.121+0000 7fe1f37fe700 1 -- 172.21.15.173:0/441701735 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe1e4025070 con 0x7fe1fc0f5410 2024-01-21T19:49:10.114 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.121+0000 7fe201169700 1 -- 172.21.15.173:0/441701735 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1fc059100 con 0x7fe1fc0f5410 2024-01-21T19:49:10.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.121+0000 7fe1f37fe700 1 --2- 172.21.15.173:0/441701735 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe1e802ff60 0x7fe1e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:10.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.121+0000 7fe1f37fe700 1 -- 172.21.15.173:0/441701735 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe1e4040c90 con 0x7fe1fc0f5410 2024-01-21T19:49:10.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.121+0000 7fe1fa59c700 1 --2- 172.21.15.173:0/441701735 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe1e802ff60 0x7fe1e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:10.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.121+0000 7fe1fa59c700 1 --2- 172.21.15.173:0/441701735 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe1e802ff60 0x7fe1e8032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fe1ec0096a0 tx=0x7fe1ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:10.118 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.126+0000 7fe1f37fe700 1 -- 172.21.15.173:0/441701735 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe1e4025950 con 0x7fe1fc0f5410 2024-01-21T19:49:10.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.331+0000 7fe201169700 1 -- 172.21.15.173:0/441701735 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1fc045ad0 con 0x7fe1fc0f5410 2024-01-21T19:49:10.325 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.332+0000 7fe1f37fe700 1 -- 172.21.15.173:0/441701735 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe1e4025950 con 0x7fe1fc0f5410 2024-01-21T19:49:10.325 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:49:10.325 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:49:10.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.334+0000 7fe201169700 1 -- 172.21.15.173:0/441701735 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe1e802ff60 msgr2=0x7fe1e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:10.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.334+0000 7fe201169700 1 --2- 172.21.15.173:0/441701735 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe1e802ff60 0x7fe1e8032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fe1ec0096a0 tx=0x7fe1ec006b40).stop 2024-01-21T19:49:10.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.334+0000 7fe201169700 1 -- 172.21.15.173:0/441701735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1fc0f5410 msgr2=0x7fe1fc109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:10.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.334+0000 7fe201169700 1 --2- 172.21.15.173:0/441701735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1fc0f5410 0x7fe1fc109450 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fe1e4014000 tx=0x7fe1e400b040).stop 2024-01-21T19:49:10.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.334+0000 7fe201169700 1 -- 172.21.15.173:0/441701735 shutdown_connections 2024-01-21T19:49:10.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.334+0000 7fe201169700 1 --2- 172.21.15.173:0/441701735 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe1e802ff60 0x7fe1e8032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:10.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.334+0000 7fe201169700 1 --2- 172.21.15.173:0/441701735 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1fc0f5410 0x7fe1fc109450 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:10.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.334+0000 7fe201169700 1 -- 172.21.15.173:0/441701735 >> 172.21.15.173:0/441701735 conn(0x7fe1fc0f0b70 msgr2=0x7fe1fc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:10.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.334+0000 7fe201169700 1 -- 172.21.15.173:0/441701735 shutdown_connections 2024-01-21T19:49:10.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:10.334+0000 7fe201169700 1 -- 172.21.15.173:0/441701735 wait complete. 2024-01-21T19:49:10.327 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:49:10.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:10 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:08.937350+0000 mgr.smithi169.espgue (mgr.14162) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:10.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:10 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:10.332758+0000 mon.smithi169 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.173:0/441701735' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:49:11.711 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:49:11.711 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:49:12.880 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.886+0000 7f1ad25f0700 1 -- 172.21.15.173:0/2743622996 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1acc060280 msgr2=0x7f1acc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:12.880 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.886+0000 7f1ad25f0700 1 --2- 172.21.15.173:0/2743622996 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1acc060280 0x7f1acc0f90f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f1ab4009fb0 tx=0x7f1ab4009b30).stop 2024-01-21T19:49:12.880 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.886+0000 7f1ad25f0700 1 -- 172.21.15.173:0/2743622996 shutdown_connections 2024-01-21T19:49:12.880 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.886+0000 7f1ad25f0700 1 --2- 172.21.15.173:0/2743622996 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1acc060280 0x7f1acc0f90f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:12.880 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.886+0000 7f1ad25f0700 1 -- 172.21.15.173:0/2743622996 >> 172.21.15.173:0/2743622996 conn(0x7f1acc0f0b70 msgr2=0x7f1acc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:12.880 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.886+0000 7f1ad25f0700 1 -- 172.21.15.173:0/2743622996 shutdown_connections 2024-01-21T19:49:12.880 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.886+0000 7f1ad25f0700 1 -- 172.21.15.173:0/2743622996 wait complete. 2024-01-21T19:49:12.880 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.887+0000 7f1ad25f0700 1 Processor -- start 2024-01-21T19:49:12.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.887+0000 7f1ad25f0700 1 -- start start 2024-01-21T19:49:12.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.887+0000 7f1ad25f0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1acc060280 0x7f1acc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:12.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.887+0000 7f1ad25f0700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc106bd0 con 0x7f1acc060280 2024-01-21T19:49:12.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.887+0000 7f1acbfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1acc060280 0x7f1acc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:12.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.887+0000 7f1acbfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1acc060280 0x7f1acc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:45748/0 (socket says 172.21.15.173:45748) 2024-01-21T19:49:12.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.887+0000 7f1acbfff700 1 -- 172.21.15.173:0/1304524158 learned_addr learned my addr 172.21.15.173:0/1304524158 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:49:12.883 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.888+0000 7f1acbfff700 1 -- 172.21.15.173:0/1304524158 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab4005040 con 0x7f1acc060280 2024-01-21T19:49:12.883 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.888+0000 7f1acbfff700 1 --2- 172.21.15.173:0/1304524158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1acc060280 0x7f1acc1066e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f1ab4013510 tx=0x7f1ab400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:12.883 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.888+0000 7f1ac8ff9700 1 -- 172.21.15.173:0/1304524158 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ab4009320 con 0x7f1acc060280 2024-01-21T19:49:12.883 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.888+0000 7f1ad25f0700 1 -- 172.21.15.173:0/1304524158 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1acc106dd0 con 0x7f1acc060280 2024-01-21T19:49:12.883 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.888+0000 7f1ac8ff9700 1 -- 172.21.15.173:0/1304524158 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ab4013f00 con 0x7f1acc060280 2024-01-21T19:49:12.883 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.888+0000 7f1ac8ff9700 1 -- 172.21.15.173:0/1304524158 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ab4014dc0 con 0x7f1acc060280 2024-01-21T19:49:12.883 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.888+0000 7f1ad25f0700 1 -- 172.21.15.173:0/1304524158 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1acc107260 con 0x7f1acc060280 2024-01-21T19:49:12.883 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.889+0000 7f1ac8ff9700 1 -- 172.21.15.173:0/1304524158 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ab4014420 con 0x7f1acc060280 2024-01-21T19:49:12.883 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.889+0000 7f1ac8ff9700 1 --2- 172.21.15.173:0/1304524158 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1ab802ffb0 0x7f1ab8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:12.883 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.889+0000 7f1ad25f0700 1 -- 172.21.15.173:0/1304524158 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1acc045ad0 con 0x7f1acc060280 2024-01-21T19:49:12.883 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.889+0000 7f1ac8ff9700 1 -- 172.21.15.173:0/1304524158 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ab4040ff0 con 0x7f1acc060280 2024-01-21T19:49:12.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.889+0000 7f1acb7fe700 1 --2- 172.21.15.173:0/1304524158 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1ab802ffb0 0x7f1ab8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:12.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.889+0000 7f1acb7fe700 1 --2- 172.21.15.173:0/1304524158 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1ab802ffb0 0x7f1ab8032470 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f1abc0096a0 tx=0x7f1abc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:12.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:12.894+0000 7f1ac8ff9700 1 -- 172.21.15.173:0/1304524158 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ab401dd50 con 0x7f1acc060280 2024-01-21T19:49:12.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:12 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:10.937603+0000 mgr.smithi169.espgue (mgr.14162) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:13.091 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:13.097+0000 7f1ad25f0700 1 -- 172.21.15.173:0/1304524158 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1acc108110 con 0x7f1acc060280 2024-01-21T19:49:13.091 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:13.098+0000 7f1ac8ff9700 1 -- 172.21.15.173:0/1304524158 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1ab4023250 con 0x7f1acc060280 2024-01-21T19:49:13.092 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:49:13.092 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:49:13.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:13.099+0000 7f1ad25f0700 1 -- 172.21.15.173:0/1304524158 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1ab802ffb0 msgr2=0x7f1ab8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:13.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:13.099+0000 7f1ad25f0700 1 --2- 172.21.15.173:0/1304524158 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1ab802ffb0 0x7f1ab8032470 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f1abc0096a0 tx=0x7f1abc006b40).stop 2024-01-21T19:49:13.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:13.099+0000 7f1ad25f0700 1 -- 172.21.15.173:0/1304524158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1acc060280 msgr2=0x7f1acc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:13.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:13.099+0000 7f1ad25f0700 1 --2- 172.21.15.173:0/1304524158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1acc060280 0x7f1acc1066e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f1ab4013510 tx=0x7f1ab400b040).stop 2024-01-21T19:49:13.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:13.099+0000 7f1ad25f0700 1 -- 172.21.15.173:0/1304524158 shutdown_connections 2024-01-21T19:49:13.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:13.099+0000 7f1ad25f0700 1 --2- 172.21.15.173:0/1304524158 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1ab802ffb0 0x7f1ab8032470 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:13.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:13.099+0000 7f1ad25f0700 1 --2- 172.21.15.173:0/1304524158 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1acc060280 0x7f1acc1066e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:13.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:13.099+0000 7f1ad25f0700 1 -- 172.21.15.173:0/1304524158 >> 172.21.15.173:0/1304524158 conn(0x7f1acc0f0b70 msgr2=0x7f1acc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:13.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:13.100+0000 7f1ad25f0700 1 -- 172.21.15.173:0/1304524158 shutdown_connections 2024-01-21T19:49:13.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:13.100+0000 7f1ad25f0700 1 -- 172.21.15.173:0/1304524158 wait complete. 2024-01-21T19:49:13.094 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:49:13.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:13 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:13.099749+0000 mon.smithi169 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.173:0/1304524158' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:49:14.460 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:49:14.460 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:49:14.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:14 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:12.937931+0000 mgr.smithi169.espgue (mgr.14162) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:15.649 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.654+0000 7f916ec79700 1 -- 172.21.15.173:0/1000384937 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91680f5410 msgr2=0x7f91680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:15.649 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.654+0000 7f916ec79700 1 --2- 172.21.15.173:0/1000384937 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91680f5410 0x7f91680f5830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f915c009fb0 tx=0x7f915c009b30).stop 2024-01-21T19:49:15.649 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.654+0000 7f916ec79700 1 -- 172.21.15.173:0/1000384937 shutdown_connections 2024-01-21T19:49:15.649 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.654+0000 7f916ec79700 1 --2- 172.21.15.173:0/1000384937 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91680f5410 0x7f91680f5830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:15.649 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.654+0000 7f916ec79700 1 -- 172.21.15.173:0/1000384937 >> 172.21.15.173:0/1000384937 conn(0x7f91680f0b70 msgr2=0x7f91680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:15.649 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.654+0000 7f916ec79700 1 -- 172.21.15.173:0/1000384937 shutdown_connections 2024-01-21T19:49:15.649 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.654+0000 7f916ec79700 1 -- 172.21.15.173:0/1000384937 wait complete. 2024-01-21T19:49:15.649 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.654+0000 7f916ec79700 1 Processor -- start 2024-01-21T19:49:15.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.654+0000 7f916ec79700 1 -- start start 2024-01-21T19:49:15.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.655+0000 7f916ec79700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91680f5410 0x7f91681055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:15.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.655+0000 7f916ec79700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9168105ac0 con 0x7f91680f5410 2024-01-21T19:49:15.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.655+0000 7f916ca15700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91680f5410 0x7f91681055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:15.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.655+0000 7f916ca15700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91680f5410 0x7f91681055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:36466/0 (socket says 172.21.15.173:36466) 2024-01-21T19:49:15.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.655+0000 7f916ca15700 1 -- 172.21.15.173:0/2281633749 learned_addr learned my addr 172.21.15.173:0/2281633749 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:49:15.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.655+0000 7f916ca15700 1 -- 172.21.15.173:0/2281633749 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915c005040 con 0x7f91680f5410 2024-01-21T19:49:15.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.655+0000 7f916ca15700 1 --2- 172.21.15.173:0/2281633749 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91680f5410 0x7f91681055d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f915c013510 tx=0x7f915c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:15.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.655+0000 7f91657fa700 1 -- 172.21.15.173:0/2281633749 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f915c009320 con 0x7f91680f5410 2024-01-21T19:49:15.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.655+0000 7f916ec79700 1 -- 172.21.15.173:0/2281633749 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9168105cc0 con 0x7f91680f5410 2024-01-21T19:49:15.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.655+0000 7f91657fa700 1 -- 172.21.15.173:0/2281633749 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f915c013f00 con 0x7f91680f5410 2024-01-21T19:49:15.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.655+0000 7f91657fa700 1 -- 172.21.15.173:0/2281633749 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f915c014d90 con 0x7f91680f5410 2024-01-21T19:49:15.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.655+0000 7f916ec79700 1 -- 172.21.15.173:0/2281633749 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9168106150 con 0x7f91680f5410 2024-01-21T19:49:15.651 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.656+0000 7f91657fa700 1 -- 172.21.15.173:0/2281633749 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f915c0263a0 con 0x7f91680f5410 2024-01-21T19:49:15.651 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.656+0000 7f916ec79700 1 -- 172.21.15.173:0/2281633749 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9168045ad0 con 0x7f91680f5410 2024-01-21T19:49:15.651 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.656+0000 7f91657fa700 1 --2- 172.21.15.173:0/2281633749 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f915802ff60 0x7f9158032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:15.651 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.656+0000 7f91657fa700 1 -- 172.21.15.173:0/2281633749 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f915c041330 con 0x7f91680f5410 2024-01-21T19:49:15.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.656+0000 7f9167fff700 1 --2- 172.21.15.173:0/2281633749 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f915802ff60 0x7f9158032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:15.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.657+0000 7f9167fff700 1 --2- 172.21.15.173:0/2281633749 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f915802ff60 0x7f9158032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f91540096a0 tx=0x7f9154006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:15.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.661+0000 7f91657fa700 1 -- 172.21.15.173:0/2281633749 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f915c0126f0 con 0x7f91680f5410 2024-01-21T19:49:15.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.866+0000 7f916ec79700 1 -- 172.21.15.173:0/2281633749 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9168107000 con 0x7f91680f5410 2024-01-21T19:49:15.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.867+0000 7f91657fa700 1 -- 172.21.15.173:0/2281633749 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f915c012330 con 0x7f91680f5410 2024-01-21T19:49:15.860 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:49:15.860 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:49:15.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.868+0000 7f916ec79700 1 -- 172.21.15.173:0/2281633749 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f915802ff60 msgr2=0x7f9158032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:15.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.868+0000 7f916ec79700 1 --2- 172.21.15.173:0/2281633749 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f915802ff60 0x7f9158032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f91540096a0 tx=0x7f9154006b40).stop 2024-01-21T19:49:15.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.869+0000 7f916ec79700 1 -- 172.21.15.173:0/2281633749 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91680f5410 msgr2=0x7f91681055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:15.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.869+0000 7f916ec79700 1 --2- 172.21.15.173:0/2281633749 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91680f5410 0x7f91681055d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f915c013510 tx=0x7f915c00b040).stop 2024-01-21T19:49:15.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.869+0000 7f916ec79700 1 -- 172.21.15.173:0/2281633749 shutdown_connections 2024-01-21T19:49:15.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.869+0000 7f916ec79700 1 --2- 172.21.15.173:0/2281633749 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f915802ff60 0x7f9158032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:15.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.869+0000 7f916ec79700 1 --2- 172.21.15.173:0/2281633749 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91680f5410 0x7f91681055d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:15.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.869+0000 7f916ec79700 1 -- 172.21.15.173:0/2281633749 >> 172.21.15.173:0/2281633749 conn(0x7f91680f0b70 msgr2=0x7f91680f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:15.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.869+0000 7f916ec79700 1 -- 172.21.15.173:0/2281633749 shutdown_connections 2024-01-21T19:49:15.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:15.869+0000 7f916ec79700 1 -- 172.21.15.173:0/2281633749 wait complete. 2024-01-21T19:49:15.863 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:49:16.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:16 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:14.938289+0000 mgr.smithi169.espgue (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:16.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:16 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:15.868469+0000 mon.smithi169 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.173:0/2281633749' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:49:17.239 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:49:17.239 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:49:18.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.394+0000 7fb44ffff700 1 -- 172.21.15.173:0/3348470814 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4500f51a0 msgr2=0x7fb4500f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:18.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.394+0000 7fb44ffff700 1 --2- 172.21.15.173:0/3348470814 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4500f51a0 0x7fb4500f55c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fb438009fb0 tx=0x7fb438009b30).stop 2024-01-21T19:49:18.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.394+0000 7fb44ffff700 1 -- 172.21.15.173:0/3348470814 shutdown_connections 2024-01-21T19:49:18.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.394+0000 7fb44ffff700 1 --2- 172.21.15.173:0/3348470814 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4500f51a0 0x7fb4500f55c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:18.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.394+0000 7fb44ffff700 1 -- 172.21.15.173:0/3348470814 >> 172.21.15.173:0/3348470814 conn(0x7fb4500f09a0 msgr2=0x7fb4500f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:18.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.394+0000 7fb44ffff700 1 -- 172.21.15.173:0/3348470814 shutdown_connections 2024-01-21T19:49:18.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.394+0000 7fb44ffff700 1 -- 172.21.15.173:0/3348470814 wait complete. 2024-01-21T19:49:18.390 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.395+0000 7fb44ffff700 1 Processor -- start 2024-01-21T19:49:18.390 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.395+0000 7fb44ffff700 1 -- start start 2024-01-21T19:49:18.390 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.395+0000 7fb44ffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4500f51a0 0x7fb4501053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:18.390 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.395+0000 7fb44ffff700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4501058a0 con 0x7fb4500f51a0 2024-01-21T19:49:18.391 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.395+0000 7fb44effd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4500f51a0 0x7fb4501053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:18.391 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.395+0000 7fb44effd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4500f51a0 0x7fb4501053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:36480/0 (socket says 172.21.15.173:36480) 2024-01-21T19:49:18.391 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.395+0000 7fb44effd700 1 -- 172.21.15.173:0/2589588931 learned_addr learned my addr 172.21.15.173:0/2589588931 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:49:18.391 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.395+0000 7fb44effd700 1 -- 172.21.15.173:0/2589588931 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb438005040 con 0x7fb4500f51a0 2024-01-21T19:49:18.391 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.395+0000 7fb44effd700 1 --2- 172.21.15.173:0/2589588931 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4500f51a0 0x7fb4501053b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fb438013510 tx=0x7fb43800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:18.391 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.396+0000 7fb4477fe700 1 -- 172.21.15.173:0/2589588931 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb438009320 con 0x7fb4500f51a0 2024-01-21T19:49:18.391 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.396+0000 7fb44ffff700 1 -- 172.21.15.173:0/2589588931 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb450105aa0 con 0x7fb4500f51a0 2024-01-21T19:49:18.391 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.396+0000 7fb4477fe700 1 -- 172.21.15.173:0/2589588931 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb438013f00 con 0x7fb4500f51a0 2024-01-21T19:49:18.391 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.396+0000 7fb4477fe700 1 -- 172.21.15.173:0/2589588931 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb438014dc0 con 0x7fb4500f51a0 2024-01-21T19:49:18.391 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.396+0000 7fb44ffff700 1 -- 172.21.15.173:0/2589588931 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb450105f30 con 0x7fb4500f51a0 2024-01-21T19:49:18.391 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.396+0000 7fb4477fe700 1 -- 172.21.15.173:0/2589588931 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb4380095f0 con 0x7fb4500f51a0 2024-01-21T19:49:18.391 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.397+0000 7fb44ffff700 1 -- 172.21.15.173:0/2589588931 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb450045ad0 con 0x7fb4500f51a0 2024-01-21T19:49:18.391 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.397+0000 7fb4477fe700 1 --2- 172.21.15.173:0/2589588931 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb43c02ff60 0x7fb43c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:18.391 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.397+0000 7fb4477fe700 1 -- 172.21.15.173:0/2589588931 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb438041040 con 0x7fb4500f51a0 2024-01-21T19:49:18.391 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.397+0000 7fb44e7fc700 1 --2- 172.21.15.173:0/2589588931 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb43c02ff60 0x7fb43c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:18.392 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.397+0000 7fb44e7fc700 1 --2- 172.21.15.173:0/2589588931 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb43c02ff60 0x7fb43c032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fb4400096a0 tx=0x7fb440006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:18.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.402+0000 7fb4477fe700 1 -- 172.21.15.173:0/2589588931 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb438014420 con 0x7fb4500f51a0 2024-01-21T19:49:18.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.607+0000 7fb44ffff700 1 -- 172.21.15.173:0/2589588931 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb450106de0 con 0x7fb4500f51a0 2024-01-21T19:49:18.602 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.608+0000 7fb4477fe700 1 -- 172.21.15.173:0/2589588931 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb438010230 con 0x7fb4500f51a0 2024-01-21T19:49:18.602 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:49:18.602 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:49:18.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.610+0000 7fb44ffff700 1 -- 172.21.15.173:0/2589588931 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb43c02ff60 msgr2=0x7fb43c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:18.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.610+0000 7fb44ffff700 1 --2- 172.21.15.173:0/2589588931 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb43c02ff60 0x7fb43c032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fb4400096a0 tx=0x7fb440006b40).stop 2024-01-21T19:49:18.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.610+0000 7fb44ffff700 1 -- 172.21.15.173:0/2589588931 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4500f51a0 msgr2=0x7fb4501053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:18.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.610+0000 7fb44ffff700 1 --2- 172.21.15.173:0/2589588931 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4500f51a0 0x7fb4501053b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fb438013510 tx=0x7fb43800b040).stop 2024-01-21T19:49:18.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.610+0000 7fb44ffff700 1 -- 172.21.15.173:0/2589588931 shutdown_connections 2024-01-21T19:49:18.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.610+0000 7fb44ffff700 1 --2- 172.21.15.173:0/2589588931 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb43c02ff60 0x7fb43c032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:18.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.610+0000 7fb44ffff700 1 --2- 172.21.15.173:0/2589588931 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb4500f51a0 0x7fb4501053b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:18.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.610+0000 7fb44ffff700 1 -- 172.21.15.173:0/2589588931 >> 172.21.15.173:0/2589588931 conn(0x7fb4500f09a0 msgr2=0x7fb4500f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:18.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.610+0000 7fb44ffff700 1 -- 172.21.15.173:0/2589588931 shutdown_connections 2024-01-21T19:49:18.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:18.610+0000 7fb44ffff700 1 -- 172.21.15.173:0/2589588931 wait complete. 2024-01-21T19:49:18.605 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:49:18.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:18 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:16.938610+0000 mgr.smithi169.espgue (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:18.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:18 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:18.609770+0000 mon.smithi169 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.173:0/2589588931' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:49:20.003 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:49:20.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:49:20.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:20 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:18.938979+0000 mgr.smithi169.espgue (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:22.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:22 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:20.939215+0000 mgr.smithi169.espgue (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:24.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:24 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:22.939553+0000 mgr.smithi169.espgue (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:25.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.169+0000 7f8866c91700 1 -- 172.21.15.173:0/3079002723 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8860060280 msgr2=0x7f88600f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:25.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.169+0000 7f8866c91700 1 --2- 172.21.15.173:0/3079002723 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8860060280 0x7f88600f90f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f8854009fb0 tx=0x7f8854009b30).stop 2024-01-21T19:49:25.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.170+0000 7f8866c91700 1 -- 172.21.15.173:0/3079002723 shutdown_connections 2024-01-21T19:49:25.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.170+0000 7f8866c91700 1 --2- 172.21.15.173:0/3079002723 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8860060280 0x7f88600f90f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:25.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.170+0000 7f8866c91700 1 -- 172.21.15.173:0/3079002723 >> 172.21.15.173:0/3079002723 conn(0x7f88600f0b70 msgr2=0x7f88600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:25.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.170+0000 7f8866c91700 1 -- 172.21.15.173:0/3079002723 shutdown_connections 2024-01-21T19:49:25.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.170+0000 7f8866c91700 1 -- 172.21.15.173:0/3079002723 wait complete. 2024-01-21T19:49:25.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.170+0000 7f8866c91700 1 Processor -- start 2024-01-21T19:49:25.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.170+0000 7f8866c91700 1 -- start start 2024-01-21T19:49:25.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.170+0000 7f8866c91700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8860060280 0x7f88600f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:25.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.170+0000 7f8866c91700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88600f4ff0 con 0x7f8860060280 2024-01-21T19:49:25.166 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.171+0000 7f8864a2d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8860060280 0x7f88600f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:25.166 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.171+0000 7f8864a2d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8860060280 0x7f88600f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:56832/0 (socket says 172.21.15.173:56832) 2024-01-21T19:49:25.166 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.171+0000 7f8864a2d700 1 -- 172.21.15.173:0/2138349164 learned_addr learned my addr 172.21.15.173:0/2138349164 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:49:25.166 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.171+0000 7f8864a2d700 1 -- 172.21.15.173:0/2138349164 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8854005040 con 0x7f8860060280 2024-01-21T19:49:25.166 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.171+0000 7f8864a2d700 1 --2- 172.21.15.173:0/2138349164 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8860060280 0x7f88600f6b70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f8854013510 tx=0x7f885400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:25.166 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.171+0000 7f885d7fa700 1 -- 172.21.15.173:0/2138349164 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8854009320 con 0x7f8860060280 2024-01-21T19:49:25.166 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.171+0000 7f885d7fa700 1 -- 172.21.15.173:0/2138349164 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8854013f00 con 0x7f8860060280 2024-01-21T19:49:25.166 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.171+0000 7f8866c91700 1 -- 172.21.15.173:0/2138349164 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88600f5190 con 0x7f8860060280 2024-01-21T19:49:25.166 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.171+0000 7f885d7fa700 1 -- 172.21.15.173:0/2138349164 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8854014dc0 con 0x7f8860060280 2024-01-21T19:49:25.166 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.171+0000 7f8866c91700 1 -- 172.21.15.173:0/2138349164 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88600f5620 con 0x7f8860060280 2024-01-21T19:49:25.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.172+0000 7f885d7fa700 1 -- 172.21.15.173:0/2138349164 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f885401d3d0 con 0x7f8860060280 2024-01-21T19:49:25.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.172+0000 7f885d7fa700 1 --2- 172.21.15.173:0/2138349164 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f885002ff60 0x7f8850032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:25.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.172+0000 7f8866c91700 1 -- 172.21.15.173:0/2138349164 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8860059100 con 0x7f8860060280 2024-01-21T19:49:25.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.172+0000 7f885d7fa700 1 -- 172.21.15.173:0/2138349164 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8854041030 con 0x7f8860060280 2024-01-21T19:49:25.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.172+0000 7f885ffff700 1 --2- 172.21.15.173:0/2138349164 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f885002ff60 0x7f8850032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:25.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.173+0000 7f885ffff700 1 --2- 172.21.15.173:0/2138349164 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f885002ff60 0x7f8850032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f884c0096a0 tx=0x7f884c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:25.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.177+0000 7f885d7fa700 1 -- 172.21.15.173:0/2138349164 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f88540095f0 con 0x7f8860060280 2024-01-21T19:49:25.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.381+0000 7f8866c91700 1 -- 172.21.15.173:0/2138349164 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88600fbf50 con 0x7f8860060280 2024-01-21T19:49:25.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.382+0000 7f885d7fa700 1 -- 172.21.15.173:0/2138349164 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8854023250 con 0x7f8860060280 2024-01-21T19:49:25.375 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:49:25.375 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:49:25.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.383+0000 7f8866c91700 1 -- 172.21.15.173:0/2138349164 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f885002ff60 msgr2=0x7f8850032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:25.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.383+0000 7f8866c91700 1 --2- 172.21.15.173:0/2138349164 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f885002ff60 0x7f8850032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f884c0096a0 tx=0x7f884c006b40).stop 2024-01-21T19:49:25.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.383+0000 7f8866c91700 1 -- 172.21.15.173:0/2138349164 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8860060280 msgr2=0x7f88600f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:25.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.383+0000 7f8866c91700 1 --2- 172.21.15.173:0/2138349164 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8860060280 0x7f88600f6b70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f8854013510 tx=0x7f885400b040).stop 2024-01-21T19:49:25.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.383+0000 7f8866c91700 1 -- 172.21.15.173:0/2138349164 shutdown_connections 2024-01-21T19:49:25.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.383+0000 7f8866c91700 1 --2- 172.21.15.173:0/2138349164 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f885002ff60 0x7f8850032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:25.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.383+0000 7f8866c91700 1 --2- 172.21.15.173:0/2138349164 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8860060280 0x7f88600f6b70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:25.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.383+0000 7f8866c91700 1 -- 172.21.15.173:0/2138349164 >> 172.21.15.173:0/2138349164 conn(0x7f88600f0b70 msgr2=0x7f88600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:25.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.383+0000 7f8866c91700 1 -- 172.21.15.173:0/2138349164 shutdown_connections 2024-01-21T19:49:25.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:25.383+0000 7f8866c91700 1 -- 172.21.15.173:0/2138349164 wait complete. 2024-01-21T19:49:25.378 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:49:25.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:25 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:25.383396+0000 mon.smithi169 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.173:0/2138349164' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:49:26.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:26 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:24.939910+0000 mgr.smithi169.espgue (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:28.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:28 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:26.940225+0000 mgr.smithi169.espgue (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:29.143 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:49:29.143 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:49:30.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:30 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:28.940510+0000 mgr.smithi169.espgue (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:31.352 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.358+0000 7f37f2901700 1 -- 172.21.15.173:0/4054957676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37ec0f5410 msgr2=0x7f37ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:31.352 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.358+0000 7f37f2901700 1 --2- 172.21.15.173:0/4054957676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37ec0f5410 0x7f37ec0f5830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f37dc009fb0 tx=0x7f37dc009b30).stop 2024-01-21T19:49:31.352 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.358+0000 7f37f2901700 1 -- 172.21.15.173:0/4054957676 shutdown_connections 2024-01-21T19:49:31.353 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.358+0000 7f37f2901700 1 --2- 172.21.15.173:0/4054957676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37ec0f5410 0x7f37ec0f5830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:31.353 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.358+0000 7f37f2901700 1 -- 172.21.15.173:0/4054957676 >> 172.21.15.173:0/4054957676 conn(0x7f37ec0f0b70 msgr2=0x7f37ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:31.353 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.358+0000 7f37f2901700 1 -- 172.21.15.173:0/4054957676 shutdown_connections 2024-01-21T19:49:31.353 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.358+0000 7f37f2901700 1 -- 172.21.15.173:0/4054957676 wait complete. 2024-01-21T19:49:31.353 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.359+0000 7f37f2901700 1 Processor -- start 2024-01-21T19:49:31.353 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.359+0000 7f37f2901700 1 -- start start 2024-01-21T19:49:31.353 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.359+0000 7f37f2901700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37ec0f5410 0x7f37ec105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:31.353 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.359+0000 7f37f2901700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec105b20 con 0x7f37ec0f5410 2024-01-21T19:49:31.353 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.359+0000 7f37ebfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37ec0f5410 0x7f37ec105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:31.353 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.359+0000 7f37ebfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37ec0f5410 0x7f37ec105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:56846/0 (socket says 172.21.15.173:56846) 2024-01-21T19:49:31.353 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.359+0000 7f37ebfff700 1 -- 172.21.15.173:0/2660263800 learned_addr learned my addr 172.21.15.173:0/2660263800 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:49:31.353 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.360+0000 7f37ebfff700 1 -- 172.21.15.173:0/2660263800 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37dc005040 con 0x7f37ec0f5410 2024-01-21T19:49:31.353 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.360+0000 7f37ebfff700 1 --2- 172.21.15.173:0/2660263800 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37ec0f5410 0x7f37ec105630 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f37dc013510 tx=0x7f37dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:31.353 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.360+0000 7f37e8ff9700 1 -- 172.21.15.173:0/2660263800 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37dc009320 con 0x7f37ec0f5410 2024-01-21T19:49:31.353 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.360+0000 7f37f2901700 1 -- 172.21.15.173:0/2660263800 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ec105d20 con 0x7f37ec0f5410 2024-01-21T19:49:31.354 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.360+0000 7f37e8ff9700 1 -- 172.21.15.173:0/2660263800 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f37dc013f00 con 0x7f37ec0f5410 2024-01-21T19:49:31.354 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.360+0000 7f37e8ff9700 1 -- 172.21.15.173:0/2660263800 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37dc014d90 con 0x7f37ec0f5410 2024-01-21T19:49:31.354 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.360+0000 7f37f2901700 1 -- 172.21.15.173:0/2660263800 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37ec1061b0 con 0x7f37ec0f5410 2024-01-21T19:49:31.354 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.361+0000 7f37e8ff9700 1 -- 172.21.15.173:0/2660263800 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f37dc0095f0 con 0x7f37ec0f5410 2024-01-21T19:49:31.355 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.361+0000 7f37f2901700 1 -- 172.21.15.173:0/2660263800 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37ec045ad0 con 0x7f37ec0f5410 2024-01-21T19:49:31.355 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.361+0000 7f37e8ff9700 1 --2- 172.21.15.173:0/2660263800 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f37d402ff60 0x7f37d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:31.355 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.361+0000 7f37e8ff9700 1 -- 172.21.15.173:0/2660263800 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f37dc041000 con 0x7f37ec0f5410 2024-01-21T19:49:31.355 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.361+0000 7f37eb7fe700 1 --2- 172.21.15.173:0/2660263800 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f37d402ff60 0x7f37d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:31.355 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.362+0000 7f37eb7fe700 1 --2- 172.21.15.173:0/2660263800 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f37d402ff60 0x7f37d4032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f37e00096a0 tx=0x7f37e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:31.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.366+0000 7f37e8ff9700 1 -- 172.21.15.173:0/2660263800 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37dc01d7f0 con 0x7f37ec0f5410 2024-01-21T19:49:31.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.569+0000 7f37f2901700 1 -- 172.21.15.173:0/2660263800 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37ec0fae60 con 0x7f37ec0f5410 2024-01-21T19:49:31.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.570+0000 7f37e8ff9700 1 -- 172.21.15.173:0/2660263800 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f37dc010230 con 0x7f37ec0f5410 2024-01-21T19:49:31.563 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:49:31.563 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:49:31.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.572+0000 7f37f2901700 1 -- 172.21.15.173:0/2660263800 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f37d402ff60 msgr2=0x7f37d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:31.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.572+0000 7f37f2901700 1 --2- 172.21.15.173:0/2660263800 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f37d402ff60 0x7f37d4032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f37e00096a0 tx=0x7f37e0006b40).stop 2024-01-21T19:49:31.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.572+0000 7f37f2901700 1 -- 172.21.15.173:0/2660263800 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37ec0f5410 msgr2=0x7f37ec105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:31.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.572+0000 7f37f2901700 1 --2- 172.21.15.173:0/2660263800 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37ec0f5410 0x7f37ec105630 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f37dc013510 tx=0x7f37dc00b040).stop 2024-01-21T19:49:31.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.572+0000 7f37f2901700 1 -- 172.21.15.173:0/2660263800 shutdown_connections 2024-01-21T19:49:31.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.572+0000 7f37f2901700 1 --2- 172.21.15.173:0/2660263800 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f37d402ff60 0x7f37d4032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:31.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.572+0000 7f37f2901700 1 --2- 172.21.15.173:0/2660263800 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f37ec0f5410 0x7f37ec105630 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:31.565 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.572+0000 7f37f2901700 1 -- 172.21.15.173:0/2660263800 >> 172.21.15.173:0/2660263800 conn(0x7f37ec0f0b70 msgr2=0x7f37ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:31.565 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.572+0000 7f37f2901700 1 -- 172.21.15.173:0/2660263800 shutdown_connections 2024-01-21T19:49:31.565 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:31.572+0000 7f37f2901700 1 -- 172.21.15.173:0/2660263800 wait complete. 2024-01-21T19:49:31.565 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:49:31.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:31.069662+0000 mon.smithi169 (mon.0) 128 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/mirror_snapshot_schedule"}]: dispatch 2024-01-21T19:49:31.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:31.073463+0000 mon.smithi169 (mon.0) 129 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/trash_purge_schedule"}]: dispatch 2024-01-21T19:49:31.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:31.570672+0000 mon.smithi169 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.173:0/2660263800' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:49:32.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:32 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:30.940851+0000 mgr.smithi169.espgue (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:32.932 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:49:32.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:49:34.159 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.165+0000 7f6ab7865700 1 -- 172.21.15.173:0/542200456 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ab00f5410 msgr2=0x7f6ab00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:34.159 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.165+0000 7f6ab7865700 1 --2- 172.21.15.173:0/542200456 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ab00f5410 0x7f6ab00f5830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f6aac009fb0 tx=0x7f6aac009b30).stop 2024-01-21T19:49:34.159 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.165+0000 7f6ab7865700 1 -- 172.21.15.173:0/542200456 shutdown_connections 2024-01-21T19:49:34.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.165+0000 7f6ab7865700 1 --2- 172.21.15.173:0/542200456 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ab00f5410 0x7f6ab00f5830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:34.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.165+0000 7f6ab7865700 1 -- 172.21.15.173:0/542200456 >> 172.21.15.173:0/542200456 conn(0x7f6ab00f0b70 msgr2=0x7f6ab00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:34.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.165+0000 7f6ab7865700 1 -- 172.21.15.173:0/542200456 shutdown_connections 2024-01-21T19:49:34.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.165+0000 7f6ab7865700 1 -- 172.21.15.173:0/542200456 wait complete. 2024-01-21T19:49:34.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.166+0000 7f6ab7865700 1 Processor -- start 2024-01-21T19:49:34.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.166+0000 7f6ab7865700 1 -- start start 2024-01-21T19:49:34.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.166+0000 7f6ab7865700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ab00f5410 0x7f6ab01093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:34.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.166+0000 7f6ab7865700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab0060280 con 0x7f6ab00f5410 2024-01-21T19:49:34.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.166+0000 7f6ab5601700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ab00f5410 0x7f6ab01093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:34.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.166+0000 7f6ab5601700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ab00f5410 0x7f6ab01093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:48626/0 (socket says 172.21.15.173:48626) 2024-01-21T19:49:34.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.166+0000 7f6ab5601700 1 -- 172.21.15.173:0/3161412318 learned_addr learned my addr 172.21.15.173:0/3161412318 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:49:34.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.167+0000 7f6ab5601700 1 -- 172.21.15.173:0/3161412318 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aac005040 con 0x7f6ab00f5410 2024-01-21T19:49:34.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.167+0000 7f6ab5601700 1 --2- 172.21.15.173:0/3161412318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ab00f5410 0x7f6ab01093f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f6aac013510 tx=0x7f6aac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:34.161 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.167+0000 7f6aa5ffb700 1 -- 172.21.15.173:0/3161412318 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6aac009320 con 0x7f6ab00f5410 2024-01-21T19:49:34.161 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.167+0000 7f6aa5ffb700 1 -- 172.21.15.173:0/3161412318 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6aac013f00 con 0x7f6ab00f5410 2024-01-21T19:49:34.161 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.167+0000 7f6ab7865700 1 -- 172.21.15.173:0/3161412318 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab0060480 con 0x7f6ab00f5410 2024-01-21T19:49:34.161 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.167+0000 7f6aa5ffb700 1 -- 172.21.15.173:0/3161412318 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6aac014d90 con 0x7f6ab00f5410 2024-01-21T19:49:34.161 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.167+0000 7f6ab7865700 1 -- 172.21.15.173:0/3161412318 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab0109b20 con 0x7f6ab00f5410 2024-01-21T19:49:34.162 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.168+0000 7f6aa5ffb700 1 -- 172.21.15.173:0/3161412318 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6aac0095f0 con 0x7f6ab00f5410 2024-01-21T19:49:34.162 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.168+0000 7f6ab7865700 1 -- 172.21.15.173:0/3161412318 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab0045ad0 con 0x7f6ab00f5410 2024-01-21T19:49:34.162 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.168+0000 7f6aa5ffb700 1 --2- 172.21.15.173:0/3161412318 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f6a9c02ff60 0x7f6a9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:34.162 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.168+0000 7f6aa5ffb700 1 -- 172.21.15.173:0/3161412318 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6aac040fc0 con 0x7f6ab00f5410 2024-01-21T19:49:34.162 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.168+0000 7f6ab4e00700 1 --2- 172.21.15.173:0/3161412318 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f6a9c02ff60 0x7f6a9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:34.162 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.169+0000 7f6ab4e00700 1 --2- 172.21.15.173:0/3161412318 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f6a9c02ff60 0x7f6a9c032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f6aa00096a0 tx=0x7f6aa0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:34.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.173+0000 7f6aa5ffb700 1 -- 172.21.15.173:0/3161412318 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6aac014420 con 0x7f6ab00f5410 2024-01-21T19:49:34.371 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.379+0000 7f6ab7865700 1 -- 172.21.15.173:0/3161412318 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6ab010a370 con 0x7f6ab00f5410 2024-01-21T19:49:34.372 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.380+0000 7f6aa5ffb700 1 -- 172.21.15.173:0/3161412318 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6aac010230 con 0x7f6ab00f5410 2024-01-21T19:49:34.372 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:49:34.372 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:49:34.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.381+0000 7f6ab7865700 1 -- 172.21.15.173:0/3161412318 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f6a9c02ff60 msgr2=0x7f6a9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:34.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.381+0000 7f6ab7865700 1 --2- 172.21.15.173:0/3161412318 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f6a9c02ff60 0x7f6a9c032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f6aa00096a0 tx=0x7f6aa0006b40).stop 2024-01-21T19:49:34.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.381+0000 7f6ab7865700 1 -- 172.21.15.173:0/3161412318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ab00f5410 msgr2=0x7f6ab01093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:34.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.381+0000 7f6ab7865700 1 --2- 172.21.15.173:0/3161412318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ab00f5410 0x7f6ab01093f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f6aac013510 tx=0x7f6aac00b040).stop 2024-01-21T19:49:34.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.381+0000 7f6ab7865700 1 -- 172.21.15.173:0/3161412318 shutdown_connections 2024-01-21T19:49:34.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.381+0000 7f6ab7865700 1 --2- 172.21.15.173:0/3161412318 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f6a9c02ff60 0x7f6a9c032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:34.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.381+0000 7f6ab7865700 1 --2- 172.21.15.173:0/3161412318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ab00f5410 0x7f6ab01093f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:34.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.381+0000 7f6ab7865700 1 -- 172.21.15.173:0/3161412318 >> 172.21.15.173:0/3161412318 conn(0x7f6ab00f0b70 msgr2=0x7f6ab00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:34.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.381+0000 7f6ab7865700 1 -- 172.21.15.173:0/3161412318 shutdown_connections 2024-01-21T19:49:34.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:34.381+0000 7f6ab7865700 1 -- 172.21.15.173:0/3161412318 wait complete. 2024-01-21T19:49:34.375 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:49:34.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:34 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:32.941219+0000 mgr.smithi169.espgue (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:34.916 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:34 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:34.380347+0000 mon.smithi169 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.173:0/3161412318' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:49:35.758 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:49:35.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:49:36.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:36 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:34.941472+0000 mgr.smithi169.espgue (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:36.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.962+0000 7f18348f0700 1 -- 172.21.15.173:0/1386789262 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18300f6510 msgr2=0x7f18300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:36.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.962+0000 7f18348f0700 1 --2- 172.21.15.173:0/1386789262 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18300f6510 0x7f18300f6930 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f1818009fb0 tx=0x7f1818009b30).stop 2024-01-21T19:49:36.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.963+0000 7f18348f0700 1 -- 172.21.15.173:0/1386789262 shutdown_connections 2024-01-21T19:49:36.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.963+0000 7f18348f0700 1 --2- 172.21.15.173:0/1386789262 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18300f6510 0x7f18300f6930 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:36.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.963+0000 7f18348f0700 1 -- 172.21.15.173:0/1386789262 >> 172.21.15.173:0/1386789262 conn(0x7f18300f1c70 msgr2=0x7f18300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:36.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.963+0000 7f18348f0700 1 -- 172.21.15.173:0/1386789262 shutdown_connections 2024-01-21T19:49:36.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.963+0000 7f18348f0700 1 -- 172.21.15.173:0/1386789262 wait complete. 2024-01-21T19:49:36.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.963+0000 7f18348f0700 1 Processor -- start 2024-01-21T19:49:36.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.963+0000 7f18348f0700 1 -- start start 2024-01-21T19:49:36.958 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.963+0000 7f18348f0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18300f6510 0x7f1830106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:36.958 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.963+0000 7f18348f0700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1830106c20 con 0x7f18300f6510 2024-01-21T19:49:36.958 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.964+0000 7f182e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18300f6510 0x7f1830106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:36.958 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.964+0000 7f182e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18300f6510 0x7f1830106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:48642/0 (socket says 172.21.15.173:48642) 2024-01-21T19:49:36.958 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.964+0000 7f182e59c700 1 -- 172.21.15.173:0/2422708022 learned_addr learned my addr 172.21.15.173:0/2422708022 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:49:36.958 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.964+0000 7f182e59c700 1 -- 172.21.15.173:0/2422708022 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1818005040 con 0x7f18300f6510 2024-01-21T19:49:36.958 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.964+0000 7f182e59c700 1 --2- 172.21.15.173:0/2422708022 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18300f6510 0x7f1830106730 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f1818014000 tx=0x7f181800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:36.958 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.964+0000 7f1826ffd700 1 -- 172.21.15.173:0/2422708022 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18180095f0 con 0x7f18300f6510 2024-01-21T19:49:36.958 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.964+0000 7f18348f0700 1 -- 172.21.15.173:0/2422708022 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1830106e20 con 0x7f18300f6510 2024-01-21T19:49:36.958 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.964+0000 7f1826ffd700 1 -- 172.21.15.173:0/2422708022 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1818007e70 con 0x7f18300f6510 2024-01-21T19:49:36.958 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.964+0000 7f1826ffd700 1 -- 172.21.15.173:0/2422708022 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1818014ea0 con 0x7f18300f6510 2024-01-21T19:49:36.958 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.964+0000 7f18348f0700 1 -- 172.21.15.173:0/2422708022 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18301072b0 con 0x7f18300f6510 2024-01-21T19:49:36.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.965+0000 7f1826ffd700 1 -- 172.21.15.173:0/2422708022 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1818025070 con 0x7f18300f6510 2024-01-21T19:49:36.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.965+0000 7f18348f0700 1 -- 172.21.15.173:0/2422708022 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1830045ad0 con 0x7f18300f6510 2024-01-21T19:49:36.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.965+0000 7f1826ffd700 1 --2- 172.21.15.173:0/2422708022 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f181c02ffb0 0x7f181c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:36.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.965+0000 7f1826ffd700 1 -- 172.21.15.173:0/2422708022 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1818040a40 con 0x7f18300f6510 2024-01-21T19:49:36.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.965+0000 7f182dd9b700 1 --2- 172.21.15.173:0/2422708022 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f181c02ffb0 0x7f181c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:36.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.966+0000 7f182dd9b700 1 --2- 172.21.15.173:0/2422708022 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f181c02ffb0 0x7f181c032470 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f18200096a0 tx=0x7f1820006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:36.962 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:36.970+0000 7f1826ffd700 1 -- 172.21.15.173:0/2422708022 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f18180079c0 con 0x7f18300f6510 2024-01-21T19:49:37.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:37.174+0000 7f18348f0700 1 -- 172.21.15.173:0/2422708022 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18300fbf60 con 0x7f18300f6510 2024-01-21T19:49:37.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:37.175+0000 7f1826ffd700 1 -- 172.21.15.173:0/2422708022 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f181800f300 con 0x7f18300f6510 2024-01-21T19:49:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:49:37.168 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:49:37.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:37.176+0000 7f18348f0700 1 -- 172.21.15.173:0/2422708022 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f181c02ffb0 msgr2=0x7f181c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:37.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:37.176+0000 7f18348f0700 1 --2- 172.21.15.173:0/2422708022 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f181c02ffb0 0x7f181c032470 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f18200096a0 tx=0x7f1820006b40).stop 2024-01-21T19:49:37.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:37.176+0000 7f18348f0700 1 -- 172.21.15.173:0/2422708022 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18300f6510 msgr2=0x7f1830106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:37.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:37.176+0000 7f18348f0700 1 --2- 172.21.15.173:0/2422708022 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18300f6510 0x7f1830106730 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f1818014000 tx=0x7f181800b040).stop 2024-01-21T19:49:37.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:37.176+0000 7f18348f0700 1 -- 172.21.15.173:0/2422708022 shutdown_connections 2024-01-21T19:49:37.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:37.176+0000 7f18348f0700 1 --2- 172.21.15.173:0/2422708022 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f181c02ffb0 0x7f181c032470 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:37.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:37.176+0000 7f18348f0700 1 --2- 172.21.15.173:0/2422708022 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18300f6510 0x7f1830106730 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:37.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:37.176+0000 7f18348f0700 1 -- 172.21.15.173:0/2422708022 >> 172.21.15.173:0/2422708022 conn(0x7f18300f1c70 msgr2=0x7f18300f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:37.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:37.176+0000 7f18348f0700 1 -- 172.21.15.173:0/2422708022 shutdown_connections 2024-01-21T19:49:37.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:37.176+0000 7f18348f0700 1 -- 172.21.15.173:0/2422708022 wait complete. 2024-01-21T19:49:37.171 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:49:38.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:37 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:37.175368+0000 mon.smithi169 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.173:0/2422708022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:49:38.581 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:49:38.581 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:49:39.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:36.941737+0000 mgr.smithi169.espgue (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:39.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.830+0000 7f2797ff2700 1 -- 172.21.15.173:0/1591341853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f27900f5410 msgr2=0x7f27900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:39.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.830+0000 7f2797ff2700 1 --2- 172.21.15.173:0/1591341853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f27900f5410 0x7f27900f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f2780009fb0 tx=0x7f2780009b30).stop 2024-01-21T19:49:39.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.830+0000 7f2797ff2700 1 -- 172.21.15.173:0/1591341853 shutdown_connections 2024-01-21T19:49:39.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.830+0000 7f2797ff2700 1 --2- 172.21.15.173:0/1591341853 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f27900f5410 0x7f27900f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:39.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.830+0000 7f2797ff2700 1 -- 172.21.15.173:0/1591341853 >> 172.21.15.173:0/1591341853 conn(0x7f27900f0b70 msgr2=0x7f27900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:39.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.830+0000 7f2797ff2700 1 -- 172.21.15.173:0/1591341853 shutdown_connections 2024-01-21T19:49:39.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.830+0000 7f2797ff2700 1 -- 172.21.15.173:0/1591341853 wait complete. 2024-01-21T19:49:39.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.831+0000 7f2797ff2700 1 Processor -- start 2024-01-21T19:49:39.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.831+0000 7f2797ff2700 1 -- start start 2024-01-21T19:49:39.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.831+0000 7f2797ff2700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f27900f5410 0x7f27901066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:39.827 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.831+0000 7f2797ff2700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2790106bc0 con 0x7f27900f5410 2024-01-21T19:49:39.827 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.831+0000 7f2795d8e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f27900f5410 0x7f27901066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:39.827 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.831+0000 7f2795d8e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f27900f5410 0x7f27901066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:48664/0 (socket says 172.21.15.173:48664) 2024-01-21T19:49:39.827 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.831+0000 7f2795d8e700 1 -- 172.21.15.173:0/2773883670 learned_addr learned my addr 172.21.15.173:0/2773883670 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:49:39.827 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.831+0000 7f2795d8e700 1 -- 172.21.15.173:0/2773883670 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2780005040 con 0x7f27900f5410 2024-01-21T19:49:39.827 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.831+0000 7f2795d8e700 1 --2- 172.21.15.173:0/2773883670 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f27900f5410 0x7f27901066d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f2780013510 tx=0x7f278000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:39.827 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.832+0000 7f27867fc700 1 -- 172.21.15.173:0/2773883670 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2780009320 con 0x7f27900f5410 2024-01-21T19:49:39.827 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.832+0000 7f2797ff2700 1 -- 172.21.15.173:0/2773883670 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2790106dc0 con 0x7f27900f5410 2024-01-21T19:49:39.827 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.832+0000 7f27867fc700 1 -- 172.21.15.173:0/2773883670 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2780013f00 con 0x7f27900f5410 2024-01-21T19:49:39.827 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.832+0000 7f27867fc700 1 -- 172.21.15.173:0/2773883670 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2780014d90 con 0x7f27900f5410 2024-01-21T19:49:39.827 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.832+0000 7f2797ff2700 1 -- 172.21.15.173:0/2773883670 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2790107250 con 0x7f27900f5410 2024-01-21T19:49:39.827 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.833+0000 7f27867fc700 1 -- 172.21.15.173:0/2773883670 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2780014420 con 0x7f27900f5410 2024-01-21T19:49:39.827 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.833+0000 7f2797ff2700 1 -- 172.21.15.173:0/2773883670 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2790045ad0 con 0x7f27900f5410 2024-01-21T19:49:39.828 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.833+0000 7f27867fc700 1 --2- 172.21.15.173:0/2773883670 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f277c02ff60 0x7f277c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:39.828 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.833+0000 7f27867fc700 1 -- 172.21.15.173:0/2773883670 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2780040fe0 con 0x7f27900f5410 2024-01-21T19:49:39.828 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.833+0000 7f279558d700 1 --2- 172.21.15.173:0/2773883670 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f277c02ff60 0x7f277c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:39.828 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.833+0000 7f279558d700 1 --2- 172.21.15.173:0/2773883670 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f277c02ff60 0x7f277c032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f278c0096a0 tx=0x7f278c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:39.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:39.838+0000 7f27867fc700 1 -- 172.21.15.173:0/2773883670 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f27800095f0 con 0x7f27900f5410 2024-01-21T19:49:40.035 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:40.043+0000 7f2797ff2700 1 -- 172.21.15.173:0/2773883670 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2790108100 con 0x7f27900f5410 2024-01-21T19:49:40.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:40.043+0000 7f27867fc700 1 -- 172.21.15.173:0/2773883670 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f278001db70 con 0x7f27900f5410 2024-01-21T19:49:40.036 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:49:40.036 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:49:40.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:40.045+0000 7f2797ff2700 1 -- 172.21.15.173:0/2773883670 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f277c02ff60 msgr2=0x7f277c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:40.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:40.045+0000 7f2797ff2700 1 --2- 172.21.15.173:0/2773883670 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f277c02ff60 0x7f277c032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f278c0096a0 tx=0x7f278c006b40).stop 2024-01-21T19:49:40.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:40.045+0000 7f2797ff2700 1 -- 172.21.15.173:0/2773883670 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f27900f5410 msgr2=0x7f27901066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:40.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:40.045+0000 7f2797ff2700 1 --2- 172.21.15.173:0/2773883670 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f27900f5410 0x7f27901066d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f2780013510 tx=0x7f278000b040).stop 2024-01-21T19:49:40.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:40.045+0000 7f2797ff2700 1 -- 172.21.15.173:0/2773883670 shutdown_connections 2024-01-21T19:49:40.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:40.045+0000 7f2797ff2700 1 --2- 172.21.15.173:0/2773883670 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f277c02ff60 0x7f277c032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:40.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:40.045+0000 7f2797ff2700 1 --2- 172.21.15.173:0/2773883670 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f27900f5410 0x7f27901066d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:40.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:40.045+0000 7f2797ff2700 1 -- 172.21.15.173:0/2773883670 >> 172.21.15.173:0/2773883670 conn(0x7f27900f0b70 msgr2=0x7f279005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:40.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:40.045+0000 7f2797ff2700 1 -- 172.21.15.173:0/2773883670 shutdown_connections 2024-01-21T19:49:40.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:40.045+0000 7f2797ff2700 1 -- 172.21.15.173:0/2773883670 wait complete. 2024-01-21T19:49:40.039 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:49:41.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:40 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:38.942025+0000 mgr.smithi169.espgue (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:41.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:40 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:40.044073+0000 mon.smithi169 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.173:0/2773883670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:49:41.430 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:49:41.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:49:42.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.607+0000 7fed349da700 1 -- 172.21.15.173:0/2536791732 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed300f51a0 msgr2=0x7fed300f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:42.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.607+0000 7fed349da700 1 --2- 172.21.15.173:0/2536791732 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed300f51a0 0x7fed300f55c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fed24009fb0 tx=0x7fed24009b30).stop 2024-01-21T19:49:42.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.608+0000 7fed349da700 1 -- 172.21.15.173:0/2536791732 shutdown_connections 2024-01-21T19:49:42.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.608+0000 7fed349da700 1 --2- 172.21.15.173:0/2536791732 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed300f51a0 0x7fed300f55c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:42.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.608+0000 7fed349da700 1 -- 172.21.15.173:0/2536791732 >> 172.21.15.173:0/2536791732 conn(0x7fed300f09a0 msgr2=0x7fed300f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:42.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.608+0000 7fed349da700 1 -- 172.21.15.173:0/2536791732 shutdown_connections 2024-01-21T19:49:42.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.608+0000 7fed349da700 1 -- 172.21.15.173:0/2536791732 wait complete. 2024-01-21T19:49:42.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.608+0000 7fed349da700 1 Processor -- start 2024-01-21T19:49:42.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.608+0000 7fed349da700 1 -- start start 2024-01-21T19:49:42.606 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.608+0000 7fed349da700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed300f51a0 0x7fed301064f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:42.606 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.608+0000 7fed349da700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed301069e0 con 0x7fed300f51a0 2024-01-21T19:49:42.606 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.609+0000 7fed2ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed300f51a0 0x7fed301064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:42.606 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.609+0000 7fed2ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed300f51a0 0x7fed301064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:48670/0 (socket says 172.21.15.173:48670) 2024-01-21T19:49:42.606 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.609+0000 7fed2ed9d700 1 -- 172.21.15.173:0/3051750086 learned_addr learned my addr 172.21.15.173:0/3051750086 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:49:42.606 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.609+0000 7fed2ed9d700 1 -- 172.21.15.173:0/3051750086 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed24005040 con 0x7fed300f51a0 2024-01-21T19:49:42.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.609+0000 7fed2ed9d700 1 --2- 172.21.15.173:0/3051750086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed300f51a0 0x7fed301064f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fed24013510 tx=0x7fed2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:42.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.609+0000 7fed177fe700 1 -- 172.21.15.173:0/3051750086 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed24009320 con 0x7fed300f51a0 2024-01-21T19:49:42.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.609+0000 7fed349da700 1 -- 172.21.15.173:0/3051750086 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed30106be0 con 0x7fed300f51a0 2024-01-21T19:49:42.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.609+0000 7fed177fe700 1 -- 172.21.15.173:0/3051750086 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed24013f00 con 0x7fed300f51a0 2024-01-21T19:49:42.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.609+0000 7fed177fe700 1 -- 172.21.15.173:0/3051750086 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed24014dc0 con 0x7fed300f51a0 2024-01-21T19:49:42.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.609+0000 7fed349da700 1 -- 172.21.15.173:0/3051750086 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed30107070 con 0x7fed300f51a0 2024-01-21T19:49:42.608 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.610+0000 7fed177fe700 1 -- 172.21.15.173:0/3051750086 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed24014420 con 0x7fed300f51a0 2024-01-21T19:49:42.608 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.610+0000 7fed177fe700 1 --2- 172.21.15.173:0/3051750086 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fed1802ff60 0x7fed18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:42.608 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.610+0000 7fed349da700 1 -- 172.21.15.173:0/3051750086 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed30045ad0 con 0x7fed300f51a0 2024-01-21T19:49:42.608 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.610+0000 7fed177fe700 1 -- 172.21.15.173:0/3051750086 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed24041070 con 0x7fed300f51a0 2024-01-21T19:49:42.608 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.610+0000 7fed2e59c700 1 --2- 172.21.15.173:0/3051750086 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fed1802ff60 0x7fed18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:42.608 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.611+0000 7fed2e59c700 1 --2- 172.21.15.173:0/3051750086 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fed1802ff60 0x7fed18032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fed200096a0 tx=0x7fed20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:42.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.615+0000 7fed177fe700 1 -- 172.21.15.173:0/3051750086 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed240095f0 con 0x7fed300f51a0 2024-01-21T19:49:42.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.822+0000 7fed349da700 1 -- 172.21.15.173:0/3051750086 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed300fabe0 con 0x7fed300f51a0 2024-01-21T19:49:42.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.822+0000 7fed177fe700 1 -- 172.21.15.173:0/3051750086 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fed2401cb10 con 0x7fed300f51a0 2024-01-21T19:49:42.815 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:49:42.815 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:49:42.816 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.824+0000 7fed349da700 1 -- 172.21.15.173:0/3051750086 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fed1802ff60 msgr2=0x7fed18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:42.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.824+0000 7fed349da700 1 --2- 172.21.15.173:0/3051750086 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fed1802ff60 0x7fed18032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fed200096a0 tx=0x7fed20006b40).stop 2024-01-21T19:49:42.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.824+0000 7fed349da700 1 -- 172.21.15.173:0/3051750086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed300f51a0 msgr2=0x7fed301064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:42.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.824+0000 7fed349da700 1 --2- 172.21.15.173:0/3051750086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed300f51a0 0x7fed301064f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fed24013510 tx=0x7fed2400b040).stop 2024-01-21T19:49:42.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.824+0000 7fed349da700 1 -- 172.21.15.173:0/3051750086 shutdown_connections 2024-01-21T19:49:42.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.824+0000 7fed349da700 1 --2- 172.21.15.173:0/3051750086 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fed1802ff60 0x7fed18032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:42.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.824+0000 7fed349da700 1 --2- 172.21.15.173:0/3051750086 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fed300f51a0 0x7fed301064f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:42.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.824+0000 7fed349da700 1 -- 172.21.15.173:0/3051750086 >> 172.21.15.173:0/3051750086 conn(0x7fed300f09a0 msgr2=0x7fed3005fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:42.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.824+0000 7fed349da700 1 -- 172.21.15.173:0/3051750086 shutdown_connections 2024-01-21T19:49:42.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:42.824+0000 7fed349da700 1 -- 172.21.15.173:0/3051750086 wait complete. 2024-01-21T19:49:42.817 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:49:43.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:40.942291+0000 mgr.smithi169.espgue (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:44.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:43 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:42.823093+0000 mon.smithi169 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.173:0/3051750086' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:49:44.202 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:49:44.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:49:45.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:44 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:42.942566+0000 mgr.smithi169.espgue (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:45.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.392+0000 7f39727d4700 1 -- 172.21.15.173:0/2218548503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f396c0f5410 msgr2=0x7f396c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:45.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.392+0000 7f39727d4700 1 --2- 172.21.15.173:0/2218548503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f396c0f5410 0x7f396c0f5830 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f395c009fb0 tx=0x7f395c009b30).stop 2024-01-21T19:49:45.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.392+0000 7f39727d4700 1 -- 172.21.15.173:0/2218548503 shutdown_connections 2024-01-21T19:49:45.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.392+0000 7f39727d4700 1 --2- 172.21.15.173:0/2218548503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f396c0f5410 0x7f396c0f5830 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:45.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.392+0000 7f39727d4700 1 -- 172.21.15.173:0/2218548503 >> 172.21.15.173:0/2218548503 conn(0x7f396c0f0b70 msgr2=0x7f396c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:45.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.392+0000 7f39727d4700 1 -- 172.21.15.173:0/2218548503 shutdown_connections 2024-01-21T19:49:45.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.392+0000 7f39727d4700 1 -- 172.21.15.173:0/2218548503 wait complete. 2024-01-21T19:49:45.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.392+0000 7f39727d4700 1 Processor -- start 2024-01-21T19:49:45.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.393+0000 7f39727d4700 1 -- start start 2024-01-21T19:49:45.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.393+0000 7f39727d4700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f396c0f5410 0x7f396c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:45.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.393+0000 7f39727d4700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c060240 con 0x7f396c0f5410 2024-01-21T19:49:45.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.393+0000 7f396bfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f396c0f5410 0x7f396c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:45.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.393+0000 7f396bfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f396c0f5410 0x7f396c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:47354/0 (socket says 172.21.15.173:47354) 2024-01-21T19:49:45.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.393+0000 7f396bfff700 1 -- 172.21.15.173:0/2416748873 learned_addr learned my addr 172.21.15.173:0/2416748873 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:49:45.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.393+0000 7f396bfff700 1 -- 172.21.15.173:0/2416748873 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395c005040 con 0x7f396c0f5410 2024-01-21T19:49:45.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.393+0000 7f396bfff700 1 --2- 172.21.15.173:0/2416748873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f396c0f5410 0x7f396c109430 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f395c013510 tx=0x7f395c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:45.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.394+0000 7f3968ff9700 1 -- 172.21.15.173:0/2416748873 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f395c009320 con 0x7f396c0f5410 2024-01-21T19:49:45.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.394+0000 7f39727d4700 1 -- 172.21.15.173:0/2416748873 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f396c060440 con 0x7f396c0f5410 2024-01-21T19:49:45.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.394+0000 7f3968ff9700 1 -- 172.21.15.173:0/2416748873 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f395c013f00 con 0x7f396c0f5410 2024-01-21T19:49:45.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.394+0000 7f3968ff9700 1 -- 172.21.15.173:0/2416748873 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f395c014d90 con 0x7f396c0f5410 2024-01-21T19:49:45.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.394+0000 7f39727d4700 1 -- 172.21.15.173:0/2416748873 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396c109ae0 con 0x7f396c0f5410 2024-01-21T19:49:45.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.394+0000 7f3968ff9700 1 -- 172.21.15.173:0/2416748873 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f395c0095f0 con 0x7f396c0f5410 2024-01-21T19:49:45.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.395+0000 7f39727d4700 1 -- 172.21.15.173:0/2416748873 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f396c045ad0 con 0x7f396c0f5410 2024-01-21T19:49:45.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.395+0000 7f3968ff9700 1 --2- 172.21.15.173:0/2416748873 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f395802ff60 0x7f3958032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:45.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.395+0000 7f3968ff9700 1 -- 172.21.15.173:0/2416748873 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f395c0410c0 con 0x7f396c0f5410 2024-01-21T19:49:45.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.395+0000 7f396b7fe700 1 --2- 172.21.15.173:0/2416748873 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f395802ff60 0x7f3958032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:45.389 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.395+0000 7f396b7fe700 1 --2- 172.21.15.173:0/2416748873 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f395802ff60 0x7f3958032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f39540096a0 tx=0x7f3954006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:45.392 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.399+0000 7f3968ff9700 1 -- 172.21.15.173:0/2416748873 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f395c014420 con 0x7f396c0f5410 2024-01-21T19:49:45.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.606+0000 7f39727d4700 1 -- 172.21.15.173:0/2416748873 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f396c0fae60 con 0x7f396c0f5410 2024-01-21T19:49:45.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.607+0000 7f3968ff9700 1 -- 172.21.15.173:0/2416748873 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f395c01d7f0 con 0x7f396c0f5410 2024-01-21T19:49:45.599 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:49:45.599 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:49:45.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.608+0000 7f39727d4700 1 -- 172.21.15.173:0/2416748873 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f395802ff60 msgr2=0x7f3958032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:45.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.608+0000 7f39727d4700 1 --2- 172.21.15.173:0/2416748873 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f395802ff60 0x7f3958032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f39540096a0 tx=0x7f3954006b40).stop 2024-01-21T19:49:45.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.608+0000 7f39727d4700 1 -- 172.21.15.173:0/2416748873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f396c0f5410 msgr2=0x7f396c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:45.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.608+0000 7f39727d4700 1 --2- 172.21.15.173:0/2416748873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f396c0f5410 0x7f396c109430 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f395c013510 tx=0x7f395c00b040).stop 2024-01-21T19:49:45.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.608+0000 7f39727d4700 1 -- 172.21.15.173:0/2416748873 shutdown_connections 2024-01-21T19:49:45.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.608+0000 7f39727d4700 1 --2- 172.21.15.173:0/2416748873 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f395802ff60 0x7f3958032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:45.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.608+0000 7f39727d4700 1 --2- 172.21.15.173:0/2416748873 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f396c0f5410 0x7f396c109430 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:45.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.608+0000 7f39727d4700 1 -- 172.21.15.173:0/2416748873 >> 172.21.15.173:0/2416748873 conn(0x7f396c0f0b70 msgr2=0x7f396c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:45.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.608+0000 7f39727d4700 1 -- 172.21.15.173:0/2416748873 shutdown_connections 2024-01-21T19:49:45.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:45.608+0000 7f39727d4700 1 -- 172.21.15.173:0/2416748873 wait complete. 2024-01-21T19:49:45.602 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:49:46.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:45 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:45.607448+0000 mon.smithi169 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.173:0/2416748873' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:49:46.995 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:49:46.995 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:49:47.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:46 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:44.942912+0000 mgr.smithi169.espgue (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:48.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.187+0000 7f9e19248700 1 -- 172.21.15.173:0/4227746469 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e14060280 msgr2=0x7f9e140f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:48.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.187+0000 7f9e19248700 1 --2- 172.21.15.173:0/4227746469 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e14060280 0x7f9e140f90f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f9dfc009fb0 tx=0x7f9dfc009b30).stop 2024-01-21T19:49:48.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.187+0000 7f9e19248700 1 -- 172.21.15.173:0/4227746469 shutdown_connections 2024-01-21T19:49:48.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.187+0000 7f9e19248700 1 --2- 172.21.15.173:0/4227746469 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e14060280 0x7f9e140f90f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:48.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.187+0000 7f9e19248700 1 -- 172.21.15.173:0/4227746469 >> 172.21.15.173:0/4227746469 conn(0x7f9e140f0b70 msgr2=0x7f9e140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:48.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.187+0000 7f9e19248700 1 -- 172.21.15.173:0/4227746469 shutdown_connections 2024-01-21T19:49:48.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.187+0000 7f9e19248700 1 -- 172.21.15.173:0/4227746469 wait complete. 2024-01-21T19:49:48.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.188+0000 7f9e19248700 1 Processor -- start 2024-01-21T19:49:48.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.188+0000 7f9e19248700 1 -- start start 2024-01-21T19:49:48.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.188+0000 7f9e19248700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e14060280 0x7f9e141066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:48.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.188+0000 7f9e19248700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e14106bb0 con 0x7f9e14060280 2024-01-21T19:49:48.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.188+0000 7f9e12d9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e14060280 0x7f9e141066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:48.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.188+0000 7f9e12d9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e14060280 0x7f9e141066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:47378/0 (socket says 172.21.15.173:47378) 2024-01-21T19:49:48.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.188+0000 7f9e12d9d700 1 -- 172.21.15.173:0/4096847994 learned_addr learned my addr 172.21.15.173:0/4096847994 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:49:48.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.188+0000 7f9e12d9d700 1 -- 172.21.15.173:0/4096847994 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dfc005040 con 0x7f9e14060280 2024-01-21T19:49:48.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.189+0000 7f9e12d9d700 1 --2- 172.21.15.173:0/4096847994 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e14060280 0x7f9e141066c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f9dfc013510 tx=0x7f9dfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:48.183 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.189+0000 7f9e0b7fe700 1 -- 172.21.15.173:0/4096847994 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dfc009320 con 0x7f9e14060280 2024-01-21T19:49:48.183 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.189+0000 7f9e0b7fe700 1 -- 172.21.15.173:0/4096847994 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9dfc013f00 con 0x7f9e14060280 2024-01-21T19:49:48.183 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.189+0000 7f9e19248700 1 -- 172.21.15.173:0/4096847994 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e14106db0 con 0x7f9e14060280 2024-01-21T19:49:48.183 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.189+0000 7f9e0b7fe700 1 -- 172.21.15.173:0/4096847994 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dfc014dc0 con 0x7f9e14060280 2024-01-21T19:49:48.183 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.189+0000 7f9e19248700 1 -- 172.21.15.173:0/4096847994 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e14107240 con 0x7f9e14060280 2024-01-21T19:49:48.183 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.190+0000 7f9e0b7fe700 1 -- 172.21.15.173:0/4096847994 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9dfc0095f0 con 0x7f9e14060280 2024-01-21T19:49:48.184 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.190+0000 7f9e19248700 1 -- 172.21.15.173:0/4096847994 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e14045ad0 con 0x7f9e14060280 2024-01-21T19:49:48.184 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.190+0000 7f9e0b7fe700 1 --2- 172.21.15.173:0/4096847994 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9e0002ff60 0x7f9e00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:48.184 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.190+0000 7f9e0b7fe700 1 -- 172.21.15.173:0/4096847994 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9dfc0410c0 con 0x7f9e14060280 2024-01-21T19:49:48.184 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.190+0000 7f9e1259c700 1 --2- 172.21.15.173:0/4096847994 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9e0002ff60 0x7f9e00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:48.184 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.190+0000 7f9e1259c700 1 --2- 172.21.15.173:0/4096847994 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9e0002ff60 0x7f9e00032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f9e040096a0 tx=0x7f9e04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:48.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.195+0000 7f9e0b7fe700 1 -- 172.21.15.173:0/4096847994 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9dfc01c790 con 0x7f9e14060280 2024-01-21T19:49:48.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.404+0000 7f9e19248700 1 -- 172.21.15.173:0/4096847994 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e141080f0 con 0x7f9e14060280 2024-01-21T19:49:48.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.405+0000 7f9e0b7fe700 1 -- 172.21.15.173:0/4096847994 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9dfc010230 con 0x7f9e14060280 2024-01-21T19:49:48.398 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:49:48.398 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:49:48.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.406+0000 7f9e19248700 1 -- 172.21.15.173:0/4096847994 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9e0002ff60 msgr2=0x7f9e00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:48.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.406+0000 7f9e19248700 1 --2- 172.21.15.173:0/4096847994 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9e0002ff60 0x7f9e00032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f9e040096a0 tx=0x7f9e04006b40).stop 2024-01-21T19:49:48.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.406+0000 7f9e19248700 1 -- 172.21.15.173:0/4096847994 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e14060280 msgr2=0x7f9e141066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:48.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.406+0000 7f9e19248700 1 --2- 172.21.15.173:0/4096847994 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e14060280 0x7f9e141066c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f9dfc013510 tx=0x7f9dfc00b040).stop 2024-01-21T19:49:48.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.407+0000 7f9e19248700 1 -- 172.21.15.173:0/4096847994 shutdown_connections 2024-01-21T19:49:48.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.407+0000 7f9e19248700 1 --2- 172.21.15.173:0/4096847994 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9e0002ff60 0x7f9e00032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:48.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.407+0000 7f9e19248700 1 --2- 172.21.15.173:0/4096847994 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9e14060280 0x7f9e141066c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:48.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.407+0000 7f9e19248700 1 -- 172.21.15.173:0/4096847994 >> 172.21.15.173:0/4096847994 conn(0x7f9e140f0b70 msgr2=0x7f9e140f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:48.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.407+0000 7f9e19248700 1 -- 172.21.15.173:0/4096847994 shutdown_connections 2024-01-21T19:49:48.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:48.407+0000 7f9e19248700 1 -- 172.21.15.173:0/4096847994 wait complete. 2024-01-21T19:49:48.400 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:49:49.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:46.943214+0000 mgr.smithi169.espgue (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:49.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:48.405791+0000 mon.smithi169 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.173:0/4096847994' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:49:49.782 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:49:49.783 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:49:50.973 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.978+0000 7f60a9b42700 1 -- 172.21.15.173:0/2696301792 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60a40f5410 msgr2=0x7f60a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:50.973 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.978+0000 7f60a9b42700 1 --2- 172.21.15.173:0/2696301792 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60a40f5410 0x7f60a40f5830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f6094009fb0 tx=0x7f6094009b30).stop 2024-01-21T19:49:50.973 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.978+0000 7f60a9b42700 1 -- 172.21.15.173:0/2696301792 shutdown_connections 2024-01-21T19:49:50.973 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.978+0000 7f60a9b42700 1 --2- 172.21.15.173:0/2696301792 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60a40f5410 0x7f60a40f5830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:50.973 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.978+0000 7f60a9b42700 1 -- 172.21.15.173:0/2696301792 >> 172.21.15.173:0/2696301792 conn(0x7f60a40f0b70 msgr2=0x7f60a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:50.973 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.978+0000 7f60a9b42700 1 -- 172.21.15.173:0/2696301792 shutdown_connections 2024-01-21T19:49:50.973 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.978+0000 7f60a9b42700 1 -- 172.21.15.173:0/2696301792 wait complete. 2024-01-21T19:49:50.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.979+0000 7f60a9b42700 1 Processor -- start 2024-01-21T19:49:50.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.979+0000 7f60a9b42700 1 -- start start 2024-01-21T19:49:50.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.979+0000 7f60a9b42700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60a40f5410 0x7f60a4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:50.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.979+0000 7f60a9b42700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a4105b20 con 0x7f60a40f5410 2024-01-21T19:49:50.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.980+0000 7f60a37fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60a40f5410 0x7f60a4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:50.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.980+0000 7f60a37fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60a40f5410 0x7f60a4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:47390/0 (socket says 172.21.15.173:47390) 2024-01-21T19:49:50.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.980+0000 7f60a37fe700 1 -- 172.21.15.173:0/551439831 learned_addr learned my addr 172.21.15.173:0/551439831 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:49:50.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.980+0000 7f60a37fe700 1 -- 172.21.15.173:0/551439831 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6094005040 con 0x7f60a40f5410 2024-01-21T19:49:50.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.980+0000 7f60a37fe700 1 --2- 172.21.15.173:0/551439831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60a40f5410 0x7f60a4105630 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f6094013510 tx=0x7f609400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:50.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.980+0000 7f608bfff700 1 -- 172.21.15.173:0/551439831 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6094009320 con 0x7f60a40f5410 2024-01-21T19:49:50.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.980+0000 7f608bfff700 1 -- 172.21.15.173:0/551439831 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6094013f00 con 0x7f60a40f5410 2024-01-21T19:49:50.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.980+0000 7f60a9b42700 1 -- 172.21.15.173:0/551439831 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60a4105d20 con 0x7f60a40f5410 2024-01-21T19:49:50.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.980+0000 7f608bfff700 1 -- 172.21.15.173:0/551439831 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6094014d90 con 0x7f60a40f5410 2024-01-21T19:49:50.975 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.980+0000 7f60a9b42700 1 -- 172.21.15.173:0/551439831 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a41061b0 con 0x7f60a40f5410 2024-01-21T19:49:50.975 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.981+0000 7f608bfff700 1 -- 172.21.15.173:0/551439831 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f60940263a0 con 0x7f60a40f5410 2024-01-21T19:49:50.976 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.981+0000 7f60a9b42700 1 -- 172.21.15.173:0/551439831 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60a4045ad0 con 0x7f60a40f5410 2024-01-21T19:49:50.976 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.981+0000 7f608bfff700 1 --2- 172.21.15.173:0/551439831 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f608c02ff60 0x7f608c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:50.976 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.981+0000 7f608bfff700 1 -- 172.21.15.173:0/551439831 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6094041300 con 0x7f60a40f5410 2024-01-21T19:49:50.976 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.982+0000 7f60a2ffd700 1 --2- 172.21.15.173:0/551439831 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f608c02ff60 0x7f608c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:50.976 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.982+0000 7f60a2ffd700 1 --2- 172.21.15.173:0/551439831 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f608c02ff60 0x7f608c032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f60980096a0 tx=0x7f6098006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:50.978 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:50.986+0000 7f608bfff700 1 -- 172.21.15.173:0/551439831 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60940126f0 con 0x7f60a40f5410 2024-01-21T19:49:51.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:50 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:48.943542+0000 mgr.smithi169.espgue (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:51.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:51.190+0000 7f60a9b42700 1 -- 172.21.15.173:0/551439831 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60a40fae60 con 0x7f60a40f5410 2024-01-21T19:49:51.183 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:51.190+0000 7f608bfff700 1 -- 172.21.15.173:0/551439831 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6094012330 con 0x7f60a40f5410 2024-01-21T19:49:51.183 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:49:51.184 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:49:51.186 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:51.192+0000 7f60a9b42700 1 -- 172.21.15.173:0/551439831 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f608c02ff60 msgr2=0x7f608c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:51.186 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:51.192+0000 7f60a9b42700 1 --2- 172.21.15.173:0/551439831 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f608c02ff60 0x7f608c032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f60980096a0 tx=0x7f6098006b40).stop 2024-01-21T19:49:51.186 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:51.192+0000 7f60a9b42700 1 -- 172.21.15.173:0/551439831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60a40f5410 msgr2=0x7f60a4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:51.186 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:51.192+0000 7f60a9b42700 1 --2- 172.21.15.173:0/551439831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60a40f5410 0x7f60a4105630 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f6094013510 tx=0x7f609400b040).stop 2024-01-21T19:49:51.186 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:51.192+0000 7f60a9b42700 1 -- 172.21.15.173:0/551439831 shutdown_connections 2024-01-21T19:49:51.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:51.192+0000 7f60a9b42700 1 --2- 172.21.15.173:0/551439831 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f608c02ff60 0x7f608c032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:51.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:51.192+0000 7f60a9b42700 1 --2- 172.21.15.173:0/551439831 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f60a40f5410 0x7f60a4105630 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:51.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:51.192+0000 7f60a9b42700 1 -- 172.21.15.173:0/551439831 >> 172.21.15.173:0/551439831 conn(0x7f60a40f0b70 msgr2=0x7f60a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:51.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:51.192+0000 7f60a9b42700 1 -- 172.21.15.173:0/551439831 shutdown_connections 2024-01-21T19:49:51.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:51.192+0000 7f60a9b42700 1 -- 172.21.15.173:0/551439831 wait complete. 2024-01-21T19:49:51.187 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:49:52.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:51 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:51.191086+0000 mon.smithi169 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.173:0/551439831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:49:52.820 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:49:52.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:49:53.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:52 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:50.943884+0000 mgr.smithi169.espgue (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:53.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:53.998+0000 7ff3388bc700 1 -- 172.21.15.173:0/3531683879 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3340f5410 msgr2=0x7ff3340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:53.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:53.998+0000 7ff3388bc700 1 --2- 172.21.15.173:0/3531683879 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3340f5410 0x7ff3340f5830 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7ff324009fb0 tx=0x7ff324009b30).stop 2024-01-21T19:49:53.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:53.998+0000 7ff3388bc700 1 -- 172.21.15.173:0/3531683879 shutdown_connections 2024-01-21T19:49:53.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:53.998+0000 7ff3388bc700 1 --2- 172.21.15.173:0/3531683879 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3340f5410 0x7ff3340f5830 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:53.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:53.998+0000 7ff3388bc700 1 -- 172.21.15.173:0/3531683879 >> 172.21.15.173:0/3531683879 conn(0x7ff3340f0b70 msgr2=0x7ff3340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:53.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:53.998+0000 7ff3388bc700 1 -- 172.21.15.173:0/3531683879 shutdown_connections 2024-01-21T19:49:53.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:53.998+0000 7ff3388bc700 1 -- 172.21.15.173:0/3531683879 wait complete. 2024-01-21T19:49:53.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:53.999+0000 7ff3388bc700 1 Processor -- start 2024-01-21T19:49:53.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:53.999+0000 7ff3388bc700 1 -- start start 2024-01-21T19:49:53.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:53.999+0000 7ff3388bc700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3340f5410 0x7ff334109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:53.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:53.999+0000 7ff3388bc700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff334060240 con 0x7ff3340f5410 2024-01-21T19:49:53.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:53.999+0000 7ff33259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3340f5410 0x7ff334109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:53.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:53.999+0000 7ff33259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3340f5410 0x7ff334109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:59228/0 (socket says 172.21.15.173:59228) 2024-01-21T19:49:53.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:53.999+0000 7ff33259c700 1 -- 172.21.15.173:0/34324898 learned_addr learned my addr 172.21.15.173:0/34324898 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:49:53.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.000+0000 7ff33259c700 1 -- 172.21.15.173:0/34324898 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff324005040 con 0x7ff3340f5410 2024-01-21T19:49:53.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.000+0000 7ff33259c700 1 --2- 172.21.15.173:0/34324898 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3340f5410 0x7ff334109430 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7ff324013510 tx=0x7ff32400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:53.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.000+0000 7ff32affd700 1 -- 172.21.15.173:0/34324898 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff324009320 con 0x7ff3340f5410 2024-01-21T19:49:53.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.000+0000 7ff3388bc700 1 -- 172.21.15.173:0/34324898 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff334060440 con 0x7ff3340f5410 2024-01-21T19:49:53.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.000+0000 7ff32affd700 1 -- 172.21.15.173:0/34324898 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff324013f00 con 0x7ff3340f5410 2024-01-21T19:49:53.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.000+0000 7ff32affd700 1 -- 172.21.15.173:0/34324898 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff324014d90 con 0x7ff3340f5410 2024-01-21T19:49:53.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.000+0000 7ff3388bc700 1 -- 172.21.15.173:0/34324898 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff334109ae0 con 0x7ff3340f5410 2024-01-21T19:49:53.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.001+0000 7ff32affd700 1 -- 172.21.15.173:0/34324898 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3240263a0 con 0x7ff3340f5410 2024-01-21T19:49:53.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.001+0000 7ff3388bc700 1 -- 172.21.15.173:0/34324898 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff334045ad0 con 0x7ff3340f5410 2024-01-21T19:49:53.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.001+0000 7ff32affd700 1 --2- 172.21.15.173:0/34324898 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff32002ff60 0x7ff320032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:53.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.001+0000 7ff32affd700 1 -- 172.21.15.173:0/34324898 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff3240412d0 con 0x7ff3340f5410 2024-01-21T19:49:53.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.001+0000 7ff331d9b700 1 --2- 172.21.15.173:0/34324898 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff32002ff60 0x7ff320032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:53.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.001+0000 7ff331d9b700 1 --2- 172.21.15.173:0/34324898 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff32002ff60 0x7ff320032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7ff31c0096a0 tx=0x7ff31c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:53.998 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.006+0000 7ff32affd700 1 -- 172.21.15.173:0/34324898 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff32401cb10 con 0x7ff3340f5410 2024-01-21T19:49:54.203 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.210+0000 7ff3388bc700 1 -- 172.21.15.173:0/34324898 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3340fae60 con 0x7ff3340f5410 2024-01-21T19:49:54.203 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.211+0000 7ff32affd700 1 -- 172.21.15.173:0/34324898 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff32401cb10 con 0x7ff3340f5410 2024-01-21T19:49:54.203 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:49:54.203 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:49:54.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.212+0000 7ff3388bc700 1 -- 172.21.15.173:0/34324898 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff32002ff60 msgr2=0x7ff320032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:54.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.212+0000 7ff3388bc700 1 --2- 172.21.15.173:0/34324898 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff32002ff60 0x7ff320032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7ff31c0096a0 tx=0x7ff31c006b40).stop 2024-01-21T19:49:54.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.212+0000 7ff3388bc700 1 -- 172.21.15.173:0/34324898 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3340f5410 msgr2=0x7ff334109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:54.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.212+0000 7ff3388bc700 1 --2- 172.21.15.173:0/34324898 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3340f5410 0x7ff334109430 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7ff324013510 tx=0x7ff32400b040).stop 2024-01-21T19:49:54.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.212+0000 7ff3388bc700 1 -- 172.21.15.173:0/34324898 shutdown_connections 2024-01-21T19:49:54.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.212+0000 7ff3388bc700 1 --2- 172.21.15.173:0/34324898 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff32002ff60 0x7ff320032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:54.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.212+0000 7ff3388bc700 1 --2- 172.21.15.173:0/34324898 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3340f5410 0x7ff334109430 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:54.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.212+0000 7ff3388bc700 1 -- 172.21.15.173:0/34324898 >> 172.21.15.173:0/34324898 conn(0x7ff3340f0b70 msgr2=0x7ff3340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:54.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.212+0000 7ff3388bc700 1 -- 172.21.15.173:0/34324898 shutdown_connections 2024-01-21T19:49:54.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:54.212+0000 7ff3388bc700 1 -- 172.21.15.173:0/34324898 wait complete. 2024-01-21T19:49:54.206 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:49:55.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:54 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:52.944252+0000 mgr.smithi169.espgue (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:55.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:54 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:54.211387+0000 mon.smithi169 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.173:0/34324898' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:49:55.591 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:49:55.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:49:56.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.881+0000 7f6910142700 1 -- 172.21.15.173:0/1896010286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69080f5410 msgr2=0x7f69080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:56.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.881+0000 7f6910142700 1 --2- 172.21.15.173:0/1896010286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69080f5410 0x7f69080f5830 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f68f8009fb0 tx=0x7f68f8009b30).stop 2024-01-21T19:49:56.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.881+0000 7f6910142700 1 -- 172.21.15.173:0/1896010286 shutdown_connections 2024-01-21T19:49:56.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.881+0000 7f6910142700 1 --2- 172.21.15.173:0/1896010286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69080f5410 0x7f69080f5830 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:56.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.881+0000 7f6910142700 1 -- 172.21.15.173:0/1896010286 >> 172.21.15.173:0/1896010286 conn(0x7f69080f0b70 msgr2=0x7f69080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:56.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.881+0000 7f6910142700 1 -- 172.21.15.173:0/1896010286 shutdown_connections 2024-01-21T19:49:56.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.881+0000 7f6910142700 1 -- 172.21.15.173:0/1896010286 wait complete. 2024-01-21T19:49:56.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.881+0000 7f6910142700 1 Processor -- start 2024-01-21T19:49:56.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.882+0000 7f6910142700 1 -- start start 2024-01-21T19:49:56.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.882+0000 7f6910142700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69080f5410 0x7f6908105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:56.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.882+0000 7f6910142700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6908105b20 con 0x7f69080f5410 2024-01-21T19:49:56.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.882+0000 7f690dede700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69080f5410 0x7f6908105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:56.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.882+0000 7f690dede700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69080f5410 0x7f6908105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:59246/0 (socket says 172.21.15.173:59246) 2024-01-21T19:49:56.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.882+0000 7f690dede700 1 -- 172.21.15.173:0/1608228695 learned_addr learned my addr 172.21.15.173:0/1608228695 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:49:56.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.882+0000 7f690dede700 1 -- 172.21.15.173:0/1608228695 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f8005040 con 0x7f69080f5410 2024-01-21T19:49:56.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.882+0000 7f690dede700 1 --2- 172.21.15.173:0/1608228695 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69080f5410 0x7f6908105630 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f68f8013510 tx=0x7f68f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:56.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.882+0000 7f68fe7fc700 1 -- 172.21.15.173:0/1608228695 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68f8009320 con 0x7f69080f5410 2024-01-21T19:49:56.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.882+0000 7f6910142700 1 -- 172.21.15.173:0/1608228695 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6908105d20 con 0x7f69080f5410 2024-01-21T19:49:56.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.882+0000 7f68fe7fc700 1 -- 172.21.15.173:0/1608228695 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68f8013f00 con 0x7f69080f5410 2024-01-21T19:49:56.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.882+0000 7f68fe7fc700 1 -- 172.21.15.173:0/1608228695 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68f8014d90 con 0x7f69080f5410 2024-01-21T19:49:56.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.883+0000 7f6910142700 1 -- 172.21.15.173:0/1608228695 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69081061b0 con 0x7f69080f5410 2024-01-21T19:49:56.878 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.883+0000 7f68fe7fc700 1 -- 172.21.15.173:0/1608228695 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68f80095f0 con 0x7f69080f5410 2024-01-21T19:49:56.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.883+0000 7f6910142700 1 -- 172.21.15.173:0/1608228695 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6908045ad0 con 0x7f69080f5410 2024-01-21T19:49:56.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.884+0000 7f68fe7fc700 1 --2- 172.21.15.173:0/1608228695 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f68f402ff60 0x7f68f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:56.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.884+0000 7f68fe7fc700 1 -- 172.21.15.173:0/1608228695 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f68f8041000 con 0x7f69080f5410 2024-01-21T19:49:56.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.884+0000 7f690d6dd700 1 --2- 172.21.15.173:0/1608228695 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f68f402ff60 0x7f68f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:56.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.884+0000 7f690d6dd700 1 --2- 172.21.15.173:0/1608228695 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f68f402ff60 0x7f68f4032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f69040096a0 tx=0x7f6904006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:56.881 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:56.888+0000 7f68fe7fc700 1 -- 172.21.15.173:0/1608228695 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f68f801d7f0 con 0x7f69080f5410 2024-01-21T19:49:57.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:57.092+0000 7f6910142700 1 -- 172.21.15.173:0/1608228695 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69080fae60 con 0x7f69080f5410 2024-01-21T19:49:57.086 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:57.092+0000 7f68fe7fc700 1 -- 172.21.15.173:0/1608228695 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f68f8010230 con 0x7f69080f5410 2024-01-21T19:49:57.086 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:49:57.086 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:49:57.088 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:57.094+0000 7f6910142700 1 -- 172.21.15.173:0/1608228695 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f68f402ff60 msgr2=0x7f68f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:57.088 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:57.094+0000 7f6910142700 1 --2- 172.21.15.173:0/1608228695 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f68f402ff60 0x7f68f4032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f69040096a0 tx=0x7f6904006b40).stop 2024-01-21T19:49:57.088 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:57.094+0000 7f6910142700 1 -- 172.21.15.173:0/1608228695 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69080f5410 msgr2=0x7f6908105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:57.088 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:57.094+0000 7f6910142700 1 --2- 172.21.15.173:0/1608228695 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69080f5410 0x7f6908105630 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f68f8013510 tx=0x7f68f800b040).stop 2024-01-21T19:49:57.088 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:57.094+0000 7f6910142700 1 -- 172.21.15.173:0/1608228695 shutdown_connections 2024-01-21T19:49:57.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:57.094+0000 7f6910142700 1 --2- 172.21.15.173:0/1608228695 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f68f402ff60 0x7f68f4032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:57.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:57.094+0000 7f6910142700 1 --2- 172.21.15.173:0/1608228695 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69080f5410 0x7f6908105630 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:57.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:57.094+0000 7f6910142700 1 -- 172.21.15.173:0/1608228695 >> 172.21.15.173:0/1608228695 conn(0x7f69080f0b70 msgr2=0x7f69080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:57.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:57.094+0000 7f6910142700 1 -- 172.21.15.173:0/1608228695 shutdown_connections 2024-01-21T19:49:57.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:57.094+0000 7f6910142700 1 -- 172.21.15.173:0/1608228695 wait complete. 2024-01-21T19:49:57.089 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:49:57.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:56 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:54.944599+0000 mgr.smithi169.espgue (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:58.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:57.094186+0000 mon.smithi169 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.173:0/1608228695' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:49:58.471 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:49:58.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:49:59.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:49:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:56.944967+0000 mgr.smithi169.espgue (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:49:59.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.679+0000 7f04ad30c700 1 -- 172.21.15.173:0/1614699664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04a80f6510 msgr2=0x7f04a80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:59.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.679+0000 7f04ad30c700 1 --2- 172.21.15.173:0/1614699664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04a80f6510 0x7f04a80f6930 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f049c009fb0 tx=0x7f049c009b30).stop 2024-01-21T19:49:59.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.679+0000 7f04ad30c700 1 -- 172.21.15.173:0/1614699664 shutdown_connections 2024-01-21T19:49:59.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.679+0000 7f04ad30c700 1 --2- 172.21.15.173:0/1614699664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04a80f6510 0x7f04a80f6930 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:59.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.679+0000 7f04ad30c700 1 -- 172.21.15.173:0/1614699664 >> 172.21.15.173:0/1614699664 conn(0x7f04a80f1c70 msgr2=0x7f04a80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:59.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.680+0000 7f04ad30c700 1 -- 172.21.15.173:0/1614699664 shutdown_connections 2024-01-21T19:49:59.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.680+0000 7f04ad30c700 1 -- 172.21.15.173:0/1614699664 wait complete. 2024-01-21T19:49:59.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.680+0000 7f04ad30c700 1 Processor -- start 2024-01-21T19:49:59.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.680+0000 7f04ad30c700 1 -- start start 2024-01-21T19:49:59.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.681+0000 7f04ad30c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04a80f6510 0x7f04a8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:59.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.681+0000 7f04ad30c700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a8106c20 con 0x7f04a80f6510 2024-01-21T19:49:59.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.681+0000 7f04a6ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04a80f6510 0x7f04a8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:59.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.681+0000 7f04a6ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04a80f6510 0x7f04a8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:59258/0 (socket says 172.21.15.173:59258) 2024-01-21T19:49:59.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.681+0000 7f04a6ffd700 1 -- 172.21.15.173:0/1139996135 learned_addr learned my addr 172.21.15.173:0/1139996135 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:49:59.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.681+0000 7f04a6ffd700 1 -- 172.21.15.173:0/1139996135 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c005040 con 0x7f04a80f6510 2024-01-21T19:49:59.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.681+0000 7f04a6ffd700 1 --2- 172.21.15.173:0/1139996135 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04a80f6510 0x7f04a8106730 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f049c014000 tx=0x7f049c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:59.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.682+0000 7f048f7fe700 1 -- 172.21.15.173:0/1139996135 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f049c0095f0 con 0x7f04a80f6510 2024-01-21T19:49:59.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.682+0000 7f048f7fe700 1 -- 172.21.15.173:0/1139996135 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f049c007e70 con 0x7f04a80f6510 2024-01-21T19:49:59.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.682+0000 7f048f7fe700 1 -- 172.21.15.173:0/1139996135 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f049c014ea0 con 0x7f04a80f6510 2024-01-21T19:49:59.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.682+0000 7f04ad30c700 1 -- 172.21.15.173:0/1139996135 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a8106e20 con 0x7f04a80f6510 2024-01-21T19:49:59.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.682+0000 7f04ad30c700 1 -- 172.21.15.173:0/1139996135 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a81072b0 con 0x7f04a80f6510 2024-01-21T19:49:59.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.683+0000 7f048f7fe700 1 -- 172.21.15.173:0/1139996135 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f049c025070 con 0x7f04a80f6510 2024-01-21T19:49:59.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.683+0000 7f04ad30c700 1 -- 172.21.15.173:0/1139996135 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a8045ad0 con 0x7f04a80f6510 2024-01-21T19:49:59.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.683+0000 7f048f7fe700 1 --2- 172.21.15.173:0/1139996135 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f049002ff60 0x7f0490032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:49:59.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.683+0000 7f048f7fe700 1 -- 172.21.15.173:0/1139996135 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f049c040950 con 0x7f04a80f6510 2024-01-21T19:49:59.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.683+0000 7f04a67fc700 1 --2- 172.21.15.173:0/1139996135 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f049002ff60 0x7f0490032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:49:59.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.683+0000 7f04a67fc700 1 --2- 172.21.15.173:0/1139996135 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f049002ff60 0x7f0490032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f04980096a0 tx=0x7f0498006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:49:59.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.688+0000 7f048f7fe700 1 -- 172.21.15.173:0/1139996135 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f049c0079c0 con 0x7f04a80f6510 2024-01-21T19:49:59.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.892+0000 7f04ad30c700 1 -- 172.21.15.173:0/1139996135 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04a80fbf60 con 0x7f04a80f6510 2024-01-21T19:49:59.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.893+0000 7f048f7fe700 1 -- 172.21.15.173:0/1139996135 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f049c00f300 con 0x7f04a80f6510 2024-01-21T19:49:59.887 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:49:59.887 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:49:59.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.895+0000 7f04ad30c700 1 -- 172.21.15.173:0/1139996135 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f049002ff60 msgr2=0x7f0490032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:59.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.895+0000 7f04ad30c700 1 --2- 172.21.15.173:0/1139996135 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f049002ff60 0x7f0490032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f04980096a0 tx=0x7f0498006b40).stop 2024-01-21T19:49:59.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.895+0000 7f04ad30c700 1 -- 172.21.15.173:0/1139996135 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04a80f6510 msgr2=0x7f04a8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:49:59.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.895+0000 7f04ad30c700 1 --2- 172.21.15.173:0/1139996135 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04a80f6510 0x7f04a8106730 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f049c014000 tx=0x7f049c00b040).stop 2024-01-21T19:49:59.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.895+0000 7f04ad30c700 1 -- 172.21.15.173:0/1139996135 shutdown_connections 2024-01-21T19:49:59.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.895+0000 7f04ad30c700 1 --2- 172.21.15.173:0/1139996135 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f049002ff60 0x7f0490032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:59.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.895+0000 7f04ad30c700 1 --2- 172.21.15.173:0/1139996135 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f04a80f6510 0x7f04a8106730 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:49:59.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.895+0000 7f04ad30c700 1 -- 172.21.15.173:0/1139996135 >> 172.21.15.173:0/1139996135 conn(0x7f04a80f1c70 msgr2=0x7f04a80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:49:59.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.895+0000 7f04ad30c700 1 -- 172.21.15.173:0/1139996135 shutdown_connections 2024-01-21T19:49:59.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:49:59.895+0000 7f04ad30c700 1 -- 172.21.15.173:0/1139996135 wait complete. 2024-01-21T19:49:59.890 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:50:01.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:00 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:49:58.945316+0000 mgr.smithi169.espgue (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:01.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:00 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:49:59.894826+0000 mon.smithi169 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.173:0/1139996135' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:50:01.280 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:50:01.280 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:50:02.507 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.513+0000 7f3cf1d60700 1 -- 172.21.15.173:0/1837583634 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3cec060140 msgr2=0x7f3cec0f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:02.507 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.513+0000 7f3cf1d60700 1 --2- 172.21.15.173:0/1837583634 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3cec060140 0x7f3cec0f8fc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f3ce0009fb0 tx=0x7f3ce0009b30).stop 2024-01-21T19:50:02.507 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.513+0000 7f3cf1d60700 1 -- 172.21.15.173:0/1837583634 shutdown_connections 2024-01-21T19:50:02.507 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.513+0000 7f3cf1d60700 1 --2- 172.21.15.173:0/1837583634 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3cec060140 0x7f3cec0f8fc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:02.507 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.513+0000 7f3cf1d60700 1 -- 172.21.15.173:0/1837583634 >> 172.21.15.173:0/1837583634 conn(0x7f3cec0f09a0 msgr2=0x7f3cec0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:02.507 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.513+0000 7f3cf1d60700 1 -- 172.21.15.173:0/1837583634 shutdown_connections 2024-01-21T19:50:02.508 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.513+0000 7f3cf1d60700 1 -- 172.21.15.173:0/1837583634 wait complete. 2024-01-21T19:50:02.508 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.513+0000 7f3cf1d60700 1 Processor -- start 2024-01-21T19:50:02.508 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.513+0000 7f3cf1d60700 1 -- start start 2024-01-21T19:50:02.508 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.514+0000 7f3cf1d60700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3cec060140 0x7f3cec106530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:02.508 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.514+0000 7f3cf1d60700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec106a20 con 0x7f3cec060140 2024-01-21T19:50:02.508 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.514+0000 7f3cf0d5e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3cec060140 0x7f3cec106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:02.508 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.514+0000 7f3cf0d5e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3cec060140 0x7f3cec106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:59282/0 (socket says 172.21.15.173:59282) 2024-01-21T19:50:02.508 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.514+0000 7f3cf0d5e700 1 -- 172.21.15.173:0/1207044836 learned_addr learned my addr 172.21.15.173:0/1207044836 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:50:02.509 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.514+0000 7f3cf0d5e700 1 -- 172.21.15.173:0/1207044836 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce0005040 con 0x7f3cec060140 2024-01-21T19:50:02.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.514+0000 7f3cf0d5e700 1 --2- 172.21.15.173:0/1207044836 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3cec060140 0x7f3cec106530 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f3ce0013510 tx=0x7f3ce000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:02.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.514+0000 7f3ce97fa700 1 -- 172.21.15.173:0/1207044836 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce0009320 con 0x7f3cec060140 2024-01-21T19:50:02.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.514+0000 7f3cf1d60700 1 -- 172.21.15.173:0/1207044836 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cec106c20 con 0x7f3cec060140 2024-01-21T19:50:02.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.514+0000 7f3ce97fa700 1 -- 172.21.15.173:0/1207044836 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ce0013f00 con 0x7f3cec060140 2024-01-21T19:50:02.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.515+0000 7f3ce97fa700 1 -- 172.21.15.173:0/1207044836 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce0014d90 con 0x7f3cec060140 2024-01-21T19:50:02.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.515+0000 7f3cf1d60700 1 -- 172.21.15.173:0/1207044836 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cec1070b0 con 0x7f3cec060140 2024-01-21T19:50:02.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.515+0000 7f3ce97fa700 1 -- 172.21.15.173:0/1207044836 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ce00095f0 con 0x7f3cec060140 2024-01-21T19:50:02.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.515+0000 7f3cf1d60700 1 -- 172.21.15.173:0/1207044836 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cec045ad0 con 0x7f3cec060140 2024-01-21T19:50:02.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.515+0000 7f3ce97fa700 1 --2- 172.21.15.173:0/1207044836 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3cdc02ff60 0x7f3cdc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:02.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.515+0000 7f3ce97fa700 1 -- 172.21.15.173:0/1207044836 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ce0040fe0 con 0x7f3cec060140 2024-01-21T19:50:02.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.516+0000 7f3cebfff700 1 --2- 172.21.15.173:0/1207044836 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3cdc02ff60 0x7f3cdc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:02.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.516+0000 7f3cebfff700 1 --2- 172.21.15.173:0/1207044836 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3cdc02ff60 0x7f3cdc032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f3cd80096a0 tx=0x7f3cd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:02.513 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.520+0000 7f3ce97fa700 1 -- 172.21.15.173:0/1207044836 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ce0010230 con 0x7f3cec060140 2024-01-21T19:50:02.716 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.723+0000 7f3cf1d60700 1 -- 172.21.15.173:0/1207044836 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3cec107f00 con 0x7f3cec060140 2024-01-21T19:50:02.717 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.723+0000 7f3ce97fa700 1 -- 172.21.15.173:0/1207044836 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3ce0010230 con 0x7f3cec060140 2024-01-21T19:50:02.717 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:50:02.717 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:50:02.718 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.725+0000 7f3cf1d60700 1 -- 172.21.15.173:0/1207044836 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3cdc02ff60 msgr2=0x7f3cdc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:02.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.725+0000 7f3cf1d60700 1 --2- 172.21.15.173:0/1207044836 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3cdc02ff60 0x7f3cdc032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f3cd80096a0 tx=0x7f3cd8006b40).stop 2024-01-21T19:50:02.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.725+0000 7f3cf1d60700 1 -- 172.21.15.173:0/1207044836 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3cec060140 msgr2=0x7f3cec106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:02.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.725+0000 7f3cf1d60700 1 --2- 172.21.15.173:0/1207044836 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3cec060140 0x7f3cec106530 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f3ce0013510 tx=0x7f3ce000b040).stop 2024-01-21T19:50:02.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.725+0000 7f3cf1d60700 1 -- 172.21.15.173:0/1207044836 shutdown_connections 2024-01-21T19:50:02.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.725+0000 7f3cf1d60700 1 --2- 172.21.15.173:0/1207044836 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3cdc02ff60 0x7f3cdc032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:02.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.725+0000 7f3cf1d60700 1 --2- 172.21.15.173:0/1207044836 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3cec060140 0x7f3cec106530 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:02.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.725+0000 7f3cf1d60700 1 -- 172.21.15.173:0/1207044836 >> 172.21.15.173:0/1207044836 conn(0x7f3cec0f09a0 msgr2=0x7f3cec0f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:02.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.725+0000 7f3cf1d60700 1 -- 172.21.15.173:0/1207044836 shutdown_connections 2024-01-21T19:50:02.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:02.725+0000 7f3cf1d60700 1 -- 172.21.15.173:0/1207044836 wait complete. 2024-01-21T19:50:02.719 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:50:03.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:02 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:00.945580+0000 mgr.smithi169.espgue (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:04.112 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:50:04.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:50:04.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:03 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:02.725059+0000 mon.smithi169 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.173:0/1207044836' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:50:05.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:04 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:02.945814+0000 mgr.smithi169.espgue (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:05.322 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.324+0000 7fb17e98d700 1 -- 172.21.15.173:0/2392204709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1780f6510 msgr2=0x7fb1780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:05.322 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.324+0000 7fb17e98d700 1 --2- 172.21.15.173:0/2392204709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1780f6510 0x7fb1780f6930 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fb168009fb0 tx=0x7fb168009b30).stop 2024-01-21T19:50:05.322 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.324+0000 7fb17e98d700 1 -- 172.21.15.173:0/2392204709 shutdown_connections 2024-01-21T19:50:05.322 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.324+0000 7fb17e98d700 1 --2- 172.21.15.173:0/2392204709 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1780f6510 0x7fb1780f6930 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:05.322 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.324+0000 7fb17e98d700 1 -- 172.21.15.173:0/2392204709 >> 172.21.15.173:0/2392204709 conn(0x7fb1780f1c70 msgr2=0x7fb1780f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:05.322 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.324+0000 7fb17e98d700 1 -- 172.21.15.173:0/2392204709 shutdown_connections 2024-01-21T19:50:05.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.324+0000 7fb17e98d700 1 -- 172.21.15.173:0/2392204709 wait complete. 2024-01-21T19:50:05.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.325+0000 7fb17e98d700 1 Processor -- start 2024-01-21T19:50:05.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.325+0000 7fb17e98d700 1 -- start start 2024-01-21T19:50:05.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.325+0000 7fb17e98d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1780f6510 0x7fb178106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:05.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.325+0000 7fb17e98d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb178106c20 con 0x7fb1780f6510 2024-01-21T19:50:05.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.325+0000 7fb177fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1780f6510 0x7fb178106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:05.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.325+0000 7fb177fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1780f6510 0x7fb178106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:43200/0 (socket says 172.21.15.173:43200) 2024-01-21T19:50:05.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.325+0000 7fb177fff700 1 -- 172.21.15.173:0/2792281969 learned_addr learned my addr 172.21.15.173:0/2792281969 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:50:05.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.326+0000 7fb177fff700 1 -- 172.21.15.173:0/2792281969 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb168005040 con 0x7fb1780f6510 2024-01-21T19:50:05.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.326+0000 7fb177fff700 1 --2- 172.21.15.173:0/2792281969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1780f6510 0x7fb178106730 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fb168013510 tx=0x7fb16800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:05.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.326+0000 7fb174ff9700 1 -- 172.21.15.173:0/2792281969 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb168009320 con 0x7fb1780f6510 2024-01-21T19:50:05.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.326+0000 7fb17e98d700 1 -- 172.21.15.173:0/2792281969 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb178106e20 con 0x7fb1780f6510 2024-01-21T19:50:05.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.326+0000 7fb174ff9700 1 -- 172.21.15.173:0/2792281969 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb168013f00 con 0x7fb1780f6510 2024-01-21T19:50:05.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.326+0000 7fb174ff9700 1 -- 172.21.15.173:0/2792281969 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb168014d90 con 0x7fb1780f6510 2024-01-21T19:50:05.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.326+0000 7fb17e98d700 1 -- 172.21.15.173:0/2792281969 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1781072b0 con 0x7fb1780f6510 2024-01-21T19:50:05.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.327+0000 7fb174ff9700 1 -- 172.21.15.173:0/2792281969 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb1680263a0 con 0x7fb1780f6510 2024-01-21T19:50:05.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.327+0000 7fb17e98d700 1 -- 172.21.15.173:0/2792281969 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb178045ad0 con 0x7fb1780f6510 2024-01-21T19:50:05.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.327+0000 7fb174ff9700 1 --2- 172.21.15.173:0/2792281969 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb16002ffb0 0x7fb160032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:05.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.327+0000 7fb174ff9700 1 -- 172.21.15.173:0/2792281969 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb1680413c0 con 0x7fb1780f6510 2024-01-21T19:50:05.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.327+0000 7fb1777fe700 1 --2- 172.21.15.173:0/2792281969 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb16002ffb0 0x7fb160032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:05.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.328+0000 7fb1777fe700 1 --2- 172.21.15.173:0/2792281969 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb16002ffb0 0x7fb160032470 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fb16c0096a0 tx=0x7fb16c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:05.325 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.332+0000 7fb174ff9700 1 -- 172.21.15.173:0/2792281969 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb16801ccf0 con 0x7fb1780f6510 2024-01-21T19:50:05.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.539+0000 7fb17e98d700 1 -- 172.21.15.173:0/2792281969 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb1780fbf60 con 0x7fb1780f6510 2024-01-21T19:50:05.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.540+0000 7fb174ff9700 1 -- 172.21.15.173:0/2792281969 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb16801cb10 con 0x7fb1780f6510 2024-01-21T19:50:05.534 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:50:05.534 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:50:05.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.542+0000 7fb17e98d700 1 -- 172.21.15.173:0/2792281969 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb16002ffb0 msgr2=0x7fb160032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:05.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.542+0000 7fb17e98d700 1 --2- 172.21.15.173:0/2792281969 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb16002ffb0 0x7fb160032470 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fb16c0096a0 tx=0x7fb16c006b40).stop 2024-01-21T19:50:05.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.542+0000 7fb17e98d700 1 -- 172.21.15.173:0/2792281969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1780f6510 msgr2=0x7fb178106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:05.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.542+0000 7fb17e98d700 1 --2- 172.21.15.173:0/2792281969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1780f6510 0x7fb178106730 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fb168013510 tx=0x7fb16800b040).stop 2024-01-21T19:50:05.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.542+0000 7fb17e98d700 1 -- 172.21.15.173:0/2792281969 shutdown_connections 2024-01-21T19:50:05.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.542+0000 7fb17e98d700 1 --2- 172.21.15.173:0/2792281969 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb16002ffb0 0x7fb160032470 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:05.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.542+0000 7fb17e98d700 1 --2- 172.21.15.173:0/2792281969 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb1780f6510 0x7fb178106730 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:05.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.542+0000 7fb17e98d700 1 -- 172.21.15.173:0/2792281969 >> 172.21.15.173:0/2792281969 conn(0x7fb1780f1c70 msgr2=0x7fb1780f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:05.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.542+0000 7fb17e98d700 1 -- 172.21.15.173:0/2792281969 shutdown_connections 2024-01-21T19:50:05.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:05.542+0000 7fb17e98d700 1 -- 172.21.15.173:0/2792281969 wait complete. 2024-01-21T19:50:05.536 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:50:06.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:05 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:05.541885+0000 mon.smithi169 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.173:0/2792281969' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:50:06.890 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:50:06.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:50:07.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:06 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:04.946200+0000 mgr.smithi169.espgue (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:08.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.111+0000 7f88d8bf9700 1 -- 172.21.15.173:0/467976235 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88d40f5410 msgr2=0x7f88d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:08.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.111+0000 7f88d8bf9700 1 --2- 172.21.15.173:0/467976235 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88d40f5410 0x7f88d40f5830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f88bc009fb0 tx=0x7f88bc009b30).stop 2024-01-21T19:50:08.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.111+0000 7f88d8bf9700 1 -- 172.21.15.173:0/467976235 shutdown_connections 2024-01-21T19:50:08.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.111+0000 7f88d8bf9700 1 --2- 172.21.15.173:0/467976235 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88d40f5410 0x7f88d40f5830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:08.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.111+0000 7f88d8bf9700 1 -- 172.21.15.173:0/467976235 >> 172.21.15.173:0/467976235 conn(0x7f88d40f0b70 msgr2=0x7f88d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:08.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.111+0000 7f88d8bf9700 1 -- 172.21.15.173:0/467976235 shutdown_connections 2024-01-21T19:50:08.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.111+0000 7f88d8bf9700 1 -- 172.21.15.173:0/467976235 wait complete. 2024-01-21T19:50:08.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.112+0000 7f88d8bf9700 1 Processor -- start 2024-01-21T19:50:08.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.112+0000 7f88d8bf9700 1 -- start start 2024-01-21T19:50:08.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.112+0000 7f88d8bf9700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88d40f5410 0x7f88d4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:08.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.112+0000 7f88d8bf9700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d4060240 con 0x7f88d40f5410 2024-01-21T19:50:08.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.112+0000 7f88d259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88d40f5410 0x7f88d4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:08.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.112+0000 7f88d259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88d40f5410 0x7f88d4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:43220/0 (socket says 172.21.15.173:43220) 2024-01-21T19:50:08.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.112+0000 7f88d259c700 1 -- 172.21.15.173:0/629559855 learned_addr learned my addr 172.21.15.173:0/629559855 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:50:08.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.113+0000 7f88d259c700 1 -- 172.21.15.173:0/629559855 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88bc005040 con 0x7f88d40f5410 2024-01-21T19:50:08.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.113+0000 7f88d259c700 1 --2- 172.21.15.173:0/629559855 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88d40f5410 0x7f88d4109430 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f88bc014000 tx=0x7f88bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:08.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.113+0000 7f88caffd700 1 -- 172.21.15.173:0/629559855 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88bc0095f0 con 0x7f88d40f5410 2024-01-21T19:50:08.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.113+0000 7f88d8bf9700 1 -- 172.21.15.173:0/629559855 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d4060440 con 0x7f88d40f5410 2024-01-21T19:50:08.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.113+0000 7f88caffd700 1 -- 172.21.15.173:0/629559855 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f88bc007e70 con 0x7f88d40f5410 2024-01-21T19:50:08.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.113+0000 7f88caffd700 1 -- 172.21.15.173:0/629559855 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88bc014ea0 con 0x7f88d40f5410 2024-01-21T19:50:08.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.113+0000 7f88d8bf9700 1 -- 172.21.15.173:0/629559855 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d4109ae0 con 0x7f88d40f5410 2024-01-21T19:50:08.109 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.114+0000 7f88caffd700 1 -- 172.21.15.173:0/629559855 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f88bc025070 con 0x7f88d40f5410 2024-01-21T19:50:08.109 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.114+0000 7f88d8bf9700 1 -- 172.21.15.173:0/629559855 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d4045ad0 con 0x7f88d40f5410 2024-01-21T19:50:08.109 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.114+0000 7f88caffd700 1 --2- 172.21.15.173:0/629559855 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f88c002ff60 0x7f88c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:08.109 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.114+0000 7f88caffd700 1 -- 172.21.15.173:0/629559855 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f88bc040960 con 0x7f88d40f5410 2024-01-21T19:50:08.109 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.114+0000 7f88d1d9b700 1 --2- 172.21.15.173:0/629559855 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f88c002ff60 0x7f88c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:08.109 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.114+0000 7f88d1d9b700 1 --2- 172.21.15.173:0/629559855 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f88c002ff60 0x7f88c0032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f88c40096a0 tx=0x7f88c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:08.112 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.119+0000 7f88caffd700 1 -- 172.21.15.173:0/629559855 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f88bc0079c0 con 0x7f88d40f5410 2024-01-21T19:50:08.318 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.325+0000 7f88d8bf9700 1 -- 172.21.15.173:0/629559855 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88d40fae60 con 0x7f88d40f5410 2024-01-21T19:50:08.319 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.325+0000 7f88caffd700 1 -- 172.21.15.173:0/629559855 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f88bc00f300 con 0x7f88d40f5410 2024-01-21T19:50:08.319 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:50:08.319 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:50:08.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.327+0000 7f88d8bf9700 1 -- 172.21.15.173:0/629559855 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f88c002ff60 msgr2=0x7f88c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:08.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.327+0000 7f88d8bf9700 1 --2- 172.21.15.173:0/629559855 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f88c002ff60 0x7f88c0032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f88c40096a0 tx=0x7f88c4006b40).stop 2024-01-21T19:50:08.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.327+0000 7f88d8bf9700 1 -- 172.21.15.173:0/629559855 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88d40f5410 msgr2=0x7f88d4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:08.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.327+0000 7f88d8bf9700 1 --2- 172.21.15.173:0/629559855 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88d40f5410 0x7f88d4109430 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f88bc014000 tx=0x7f88bc00b040).stop 2024-01-21T19:50:08.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.327+0000 7f88d8bf9700 1 -- 172.21.15.173:0/629559855 shutdown_connections 2024-01-21T19:50:08.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.327+0000 7f88d8bf9700 1 --2- 172.21.15.173:0/629559855 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f88c002ff60 0x7f88c0032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:08.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.327+0000 7f88d8bf9700 1 --2- 172.21.15.173:0/629559855 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f88d40f5410 0x7f88d4109430 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:08.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.327+0000 7f88d8bf9700 1 -- 172.21.15.173:0/629559855 >> 172.21.15.173:0/629559855 conn(0x7f88d40f0b70 msgr2=0x7f88d40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:08.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.327+0000 7f88d8bf9700 1 -- 172.21.15.173:0/629559855 shutdown_connections 2024-01-21T19:50:08.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:08.327+0000 7f88d8bf9700 1 -- 172.21.15.173:0/629559855 wait complete. 2024-01-21T19:50:08.322 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:50:09.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:08 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:06.946491+0000 mgr.smithi169.espgue (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:09.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:08 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:08.327027+0000 mon.smithi169 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.173:0/629559855' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:50:09.710 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:50:09.710 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:50:10.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.926+0000 7fab307a4700 1 -- 172.21.15.173:0/3634109451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab280f6510 msgr2=0x7fab280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:10.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.926+0000 7fab307a4700 1 --2- 172.21.15.173:0/3634109451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab280f6510 0x7fab280f6930 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fab18009fb0 tx=0x7fab18009b30).stop 2024-01-21T19:50:10.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.926+0000 7fab307a4700 1 -- 172.21.15.173:0/3634109451 shutdown_connections 2024-01-21T19:50:10.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.926+0000 7fab307a4700 1 --2- 172.21.15.173:0/3634109451 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab280f6510 0x7fab280f6930 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:10.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.926+0000 7fab307a4700 1 -- 172.21.15.173:0/3634109451 >> 172.21.15.173:0/3634109451 conn(0x7fab280f1c70 msgr2=0x7fab280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:10.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.926+0000 7fab307a4700 1 -- 172.21.15.173:0/3634109451 shutdown_connections 2024-01-21T19:50:10.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.926+0000 7fab307a4700 1 -- 172.21.15.173:0/3634109451 wait complete. 2024-01-21T19:50:10.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.927+0000 7fab307a4700 1 Processor -- start 2024-01-21T19:50:10.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.927+0000 7fab307a4700 1 -- start start 2024-01-21T19:50:10.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.927+0000 7fab307a4700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab280f6510 0x7fab2806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:10.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.927+0000 7fab307a4700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab28069e20 con 0x7fab280f6510 2024-01-21T19:50:10.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.927+0000 7fab2e540700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab280f6510 0x7fab2806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:10.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.927+0000 7fab2e540700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab280f6510 0x7fab2806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:43232/0 (socket says 172.21.15.173:43232) 2024-01-21T19:50:10.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.927+0000 7fab2e540700 1 -- 172.21.15.173:0/2867003582 learned_addr learned my addr 172.21.15.173:0/2867003582 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:50:10.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.927+0000 7fab2e540700 1 -- 172.21.15.173:0/2867003582 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab18005040 con 0x7fab280f6510 2024-01-21T19:50:10.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.928+0000 7fab2e540700 1 --2- 172.21.15.173:0/2867003582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab280f6510 0x7fab2806b9a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fab18013510 tx=0x7fab1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:10.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.928+0000 7fab1effd700 1 -- 172.21.15.173:0/2867003582 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab18009320 con 0x7fab280f6510 2024-01-21T19:50:10.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.928+0000 7fab307a4700 1 -- 172.21.15.173:0/2867003582 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab28069fc0 con 0x7fab280f6510 2024-01-21T19:50:10.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.928+0000 7fab307a4700 1 -- 172.21.15.173:0/2867003582 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab2806a450 con 0x7fab280f6510 2024-01-21T19:50:10.923 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.928+0000 7fab1effd700 1 -- 172.21.15.173:0/2867003582 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab18013f00 con 0x7fab280f6510 2024-01-21T19:50:10.923 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.928+0000 7fab1effd700 1 -- 172.21.15.173:0/2867003582 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab18014d20 con 0x7fab280f6510 2024-01-21T19:50:10.923 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.929+0000 7fab307a4700 1 -- 172.21.15.173:0/2867003582 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab10004fa0 con 0x7fab280f6510 2024-01-21T19:50:10.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.929+0000 7fab1effd700 1 -- 172.21.15.173:0/2867003582 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fab180095f0 con 0x7fab280f6510 2024-01-21T19:50:10.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.929+0000 7fab1effd700 1 --2- 172.21.15.173:0/2867003582 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fab1402ff60 0x7fab14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:10.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.929+0000 7fab1effd700 1 -- 172.21.15.173:0/2867003582 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fab18040f60 con 0x7fab280f6510 2024-01-21T19:50:10.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.929+0000 7fab2dd3f700 1 --2- 172.21.15.173:0/2867003582 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fab1402ff60 0x7fab14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:10.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.929+0000 7fab2dd3f700 1 --2- 172.21.15.173:0/2867003582 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fab1402ff60 0x7fab14032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fab240096a0 tx=0x7fab24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:10.926 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:10.933+0000 7fab1effd700 1 -- 172.21.15.173:0/2867003582 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab1806d2b0 con 0x7fab280f6510 2024-01-21T19:50:11.130 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:11.136+0000 7fab307a4700 1 -- 172.21.15.173:0/2867003582 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab10005670 con 0x7fab280f6510 2024-01-21T19:50:11.130 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:11.137+0000 7fab1effd700 1 -- 172.21.15.173:0/2867003582 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fab18014420 con 0x7fab280f6510 2024-01-21T19:50:11.131 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:50:11.131 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:50:11.132 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:11.139+0000 7fab307a4700 1 -- 172.21.15.173:0/2867003582 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fab1402ff60 msgr2=0x7fab14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:11.132 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:11.139+0000 7fab307a4700 1 --2- 172.21.15.173:0/2867003582 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fab1402ff60 0x7fab14032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fab240096a0 tx=0x7fab24006b40).stop 2024-01-21T19:50:11.132 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:11.139+0000 7fab307a4700 1 -- 172.21.15.173:0/2867003582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab280f6510 msgr2=0x7fab2806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:11.132 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:11.139+0000 7fab307a4700 1 --2- 172.21.15.173:0/2867003582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab280f6510 0x7fab2806b9a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fab18013510 tx=0x7fab1800b040).stop 2024-01-21T19:50:11.132 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:11.139+0000 7fab307a4700 1 -- 172.21.15.173:0/2867003582 shutdown_connections 2024-01-21T19:50:11.133 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:11.139+0000 7fab307a4700 1 --2- 172.21.15.173:0/2867003582 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fab1402ff60 0x7fab14032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:11.133 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:11.139+0000 7fab307a4700 1 --2- 172.21.15.173:0/2867003582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab280f6510 0x7fab2806b9a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:11.133 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:11.139+0000 7fab307a4700 1 -- 172.21.15.173:0/2867003582 >> 172.21.15.173:0/2867003582 conn(0x7fab280f1c70 msgr2=0x7fab280f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:11.133 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:11.139+0000 7fab307a4700 1 -- 172.21.15.173:0/2867003582 shutdown_connections 2024-01-21T19:50:11.133 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:11.139+0000 7fab307a4700 1 -- 172.21.15.173:0/2867003582 wait complete. 2024-01-21T19:50:11.133 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:50:11.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:10 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:08.946779+0000 mgr.smithi169.espgue (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:12.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:11 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:11.138647+0000 mon.smithi169 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.173:0/2867003582' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:50:12.534 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:50:12.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:50:13.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:12 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:10.947184+0000 mgr.smithi169.espgue (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:13.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.741+0000 7f50f51c7700 1 -- 172.21.15.173:0/561301084 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f50f00f47c0 msgr2=0x7f50f00f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:13.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.741+0000 7f50f51c7700 1 --2- 172.21.15.173:0/561301084 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f50f00f47c0 0x7f50f00f4be0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f50e0009fb0 tx=0x7f50e0009b30).stop 2024-01-21T19:50:13.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.741+0000 7f50f51c7700 1 -- 172.21.15.173:0/561301084 shutdown_connections 2024-01-21T19:50:13.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.741+0000 7f50f51c7700 1 --2- 172.21.15.173:0/561301084 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f50f00f47c0 0x7f50f00f4be0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:13.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.741+0000 7f50f51c7700 1 -- 172.21.15.173:0/561301084 >> 172.21.15.173:0/561301084 conn(0x7f50f00f02e0 msgr2=0x7f50f00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:13.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.741+0000 7f50f51c7700 1 -- 172.21.15.173:0/561301084 shutdown_connections 2024-01-21T19:50:13.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.741+0000 7f50f51c7700 1 -- 172.21.15.173:0/561301084 wait complete. 2024-01-21T19:50:13.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.741+0000 7f50f51c7700 1 Processor -- start 2024-01-21T19:50:13.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.741+0000 7f50f51c7700 1 -- start start 2024-01-21T19:50:13.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.742+0000 7f50f51c7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f50f00f47c0 0x7f50f00f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:13.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.742+0000 7f50f51c7700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f00f57f0 con 0x7f50f00f47c0 2024-01-21T19:50:13.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.742+0000 7f50eed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f50f00f47c0 0x7f50f00f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:13.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.742+0000 7f50eed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f50f00f47c0 0x7f50f00f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:37596/0 (socket says 172.21.15.173:37596) 2024-01-21T19:50:13.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.742+0000 7f50eed9d700 1 -- 172.21.15.173:0/1291437026 learned_addr learned my addr 172.21.15.173:0/1291437026 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:50:13.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.742+0000 7f50eed9d700 1 -- 172.21.15.173:0/1291437026 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e0005040 con 0x7f50f00f47c0 2024-01-21T19:50:13.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.742+0000 7f50eed9d700 1 --2- 172.21.15.173:0/1291437026 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f50f00f47c0 0x7f50f00f7370 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f50e0013510 tx=0x7f50e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:13.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.742+0000 7f50d77fe700 1 -- 172.21.15.173:0/1291437026 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50e0009320 con 0x7f50f00f47c0 2024-01-21T19:50:13.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.742+0000 7f50d77fe700 1 -- 172.21.15.173:0/1291437026 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f50e0013f00 con 0x7f50f00f47c0 2024-01-21T19:50:13.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.742+0000 7f50f51c7700 1 -- 172.21.15.173:0/1291437026 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f00f5990 con 0x7f50f00f47c0 2024-01-21T19:50:13.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.742+0000 7f50d77fe700 1 -- 172.21.15.173:0/1291437026 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50e0014d90 con 0x7f50f00f47c0 2024-01-21T19:50:13.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.743+0000 7f50f51c7700 1 -- 172.21.15.173:0/1291437026 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50f00f5e20 con 0x7f50f00f47c0 2024-01-21T19:50:13.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.743+0000 7f50f51c7700 1 -- 172.21.15.173:0/1291437026 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50f0059100 con 0x7f50f00f47c0 2024-01-21T19:50:13.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.744+0000 7f50d77fe700 1 -- 172.21.15.173:0/1291437026 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50e00095f0 con 0x7f50f00f47c0 2024-01-21T19:50:13.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.744+0000 7f50d77fe700 1 --2- 172.21.15.173:0/1291437026 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f50d802ff60 0x7f50d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:13.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.744+0000 7f50d77fe700 1 -- 172.21.15.173:0/1291437026 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f50e001b920 con 0x7f50f00f47c0 2024-01-21T19:50:13.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.744+0000 7f50ee59c700 1 --2- 172.21.15.173:0/1291437026 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f50d802ff60 0x7f50d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:13.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.744+0000 7f50ee59c700 1 --2- 172.21.15.173:0/1291437026 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f50d802ff60 0x7f50d8032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f50e40096a0 tx=0x7f50e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:13.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.748+0000 7f50d77fe700 1 -- 172.21.15.173:0/1291437026 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50e0014420 con 0x7f50f00f47c0 2024-01-21T19:50:13.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.953+0000 7f50f51c7700 1 -- 172.21.15.173:0/1291437026 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50f00fbf60 con 0x7f50f00f47c0 2024-01-21T19:50:13.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.953+0000 7f50d77fe700 1 -- 172.21.15.173:0/1291437026 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f50e001b740 con 0x7f50f00f47c0 2024-01-21T19:50:13.947 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:50:13.947 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:50:13.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.955+0000 7f50f51c7700 1 -- 172.21.15.173:0/1291437026 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f50d802ff60 msgr2=0x7f50d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:13.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.955+0000 7f50f51c7700 1 --2- 172.21.15.173:0/1291437026 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f50d802ff60 0x7f50d8032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f50e40096a0 tx=0x7f50e4006b40).stop 2024-01-21T19:50:13.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.955+0000 7f50f51c7700 1 -- 172.21.15.173:0/1291437026 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f50f00f47c0 msgr2=0x7f50f00f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:13.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.955+0000 7f50f51c7700 1 --2- 172.21.15.173:0/1291437026 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f50f00f47c0 0x7f50f00f7370 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f50e0013510 tx=0x7f50e000b040).stop 2024-01-21T19:50:13.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.955+0000 7f50f51c7700 1 -- 172.21.15.173:0/1291437026 shutdown_connections 2024-01-21T19:50:13.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.955+0000 7f50f51c7700 1 --2- 172.21.15.173:0/1291437026 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f50d802ff60 0x7f50d8032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:13.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.955+0000 7f50f51c7700 1 --2- 172.21.15.173:0/1291437026 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f50f00f47c0 0x7f50f00f7370 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:13.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.955+0000 7f50f51c7700 1 -- 172.21.15.173:0/1291437026 >> 172.21.15.173:0/1291437026 conn(0x7f50f00f02e0 msgr2=0x7f50f00f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:13.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.955+0000 7f50f51c7700 1 -- 172.21.15.173:0/1291437026 shutdown_connections 2024-01-21T19:50:13.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:13.955+0000 7f50f51c7700 1 -- 172.21.15.173:0/1291437026 wait complete. 2024-01-21T19:50:13.949 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:50:15.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:14 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:12.947535+0000 mgr.smithi169.espgue (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:15.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:14 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:13.955198+0000 mon.smithi169 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.173:0/1291437026' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:50:15.338 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:50:15.338 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:50:16.559 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.565+0000 7f870b9ba700 1 -- 172.21.15.173:0/3074170526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87040f53f0 msgr2=0x7f87040f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:16.560 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.565+0000 7f870b9ba700 1 --2- 172.21.15.173:0/3074170526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87040f53f0 0x7f87040f5810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f86f4009fb0 tx=0x7f86f4009b30).stop 2024-01-21T19:50:16.560 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.565+0000 7f870b9ba700 1 -- 172.21.15.173:0/3074170526 shutdown_connections 2024-01-21T19:50:16.560 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.565+0000 7f870b9ba700 1 --2- 172.21.15.173:0/3074170526 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87040f53f0 0x7f87040f5810 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:16.560 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.565+0000 7f870b9ba700 1 -- 172.21.15.173:0/3074170526 >> 172.21.15.173:0/3074170526 conn(0x7f87040f0b70 msgr2=0x7f87040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:16.560 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.565+0000 7f870b9ba700 1 -- 172.21.15.173:0/3074170526 shutdown_connections 2024-01-21T19:50:16.560 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.565+0000 7f870b9ba700 1 -- 172.21.15.173:0/3074170526 wait complete. 2024-01-21T19:50:16.560 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.566+0000 7f870b9ba700 1 Processor -- start 2024-01-21T19:50:16.560 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.566+0000 7f870b9ba700 1 -- start start 2024-01-21T19:50:16.560 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.566+0000 7f870b9ba700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87040f53f0 0x7f87041055b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:16.560 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.566+0000 7f870b9ba700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8704105aa0 con 0x7f87040f53f0 2024-01-21T19:50:16.560 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.566+0000 7f8709756700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87040f53f0 0x7f87041055b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:16.560 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.566+0000 7f8709756700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87040f53f0 0x7f87041055b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:37618/0 (socket says 172.21.15.173:37618) 2024-01-21T19:50:16.560 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.566+0000 7f8709756700 1 -- 172.21.15.173:0/3076100890 learned_addr learned my addr 172.21.15.173:0/3076100890 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:50:16.561 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.566+0000 7f8709756700 1 -- 172.21.15.173:0/3076100890 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f4005040 con 0x7f87040f53f0 2024-01-21T19:50:16.561 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.567+0000 7f8709756700 1 --2- 172.21.15.173:0/3076100890 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87040f53f0 0x7f87041055b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f86f4013510 tx=0x7f86f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:16.561 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.567+0000 7f86f9ffb700 1 -- 172.21.15.173:0/3076100890 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86f4009320 con 0x7f87040f53f0 2024-01-21T19:50:16.561 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.567+0000 7f870b9ba700 1 -- 172.21.15.173:0/3076100890 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8704105ca0 con 0x7f87040f53f0 2024-01-21T19:50:16.561 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.567+0000 7f86f9ffb700 1 -- 172.21.15.173:0/3076100890 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86f4013f00 con 0x7f87040f53f0 2024-01-21T19:50:16.561 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.567+0000 7f86f9ffb700 1 -- 172.21.15.173:0/3076100890 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86f4014d90 con 0x7f87040f53f0 2024-01-21T19:50:16.561 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.567+0000 7f870b9ba700 1 -- 172.21.15.173:0/3076100890 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8704106130 con 0x7f87040f53f0 2024-01-21T19:50:16.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.568+0000 7f86f9ffb700 1 -- 172.21.15.173:0/3076100890 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f86f40095f0 con 0x7f87040f53f0 2024-01-21T19:50:16.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.568+0000 7f870b9ba700 1 -- 172.21.15.173:0/3076100890 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8704045ad0 con 0x7f87040f53f0 2024-01-21T19:50:16.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.568+0000 7f86f9ffb700 1 --2- 172.21.15.173:0/3076100890 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f86f002ff60 0x7f86f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:16.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.568+0000 7f86f9ffb700 1 -- 172.21.15.173:0/3076100890 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f86f4041000 con 0x7f87040f53f0 2024-01-21T19:50:16.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.568+0000 7f8708f55700 1 --2- 172.21.15.173:0/3076100890 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f86f002ff60 0x7f86f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:16.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.569+0000 7f8708f55700 1 --2- 172.21.15.173:0/3076100890 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f86f002ff60 0x7f86f0032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f87000096a0 tx=0x7f8700006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:16.565 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.573+0000 7f86f9ffb700 1 -- 172.21.15.173:0/3076100890 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f86f4014420 con 0x7f87040f53f0 2024-01-21T19:50:16.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.776+0000 7f870b9ba700 1 -- 172.21.15.173:0/3076100890 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8704106fe0 con 0x7f87040f53f0 2024-01-21T19:50:16.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.777+0000 7f86f9ffb700 1 -- 172.21.15.173:0/3076100890 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f86f401d7f0 con 0x7f87040f53f0 2024-01-21T19:50:16.769 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:50:16.769 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:50:16.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.778+0000 7f870b9ba700 1 -- 172.21.15.173:0/3076100890 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f86f002ff60 msgr2=0x7f86f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:16.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.778+0000 7f870b9ba700 1 --2- 172.21.15.173:0/3076100890 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f86f002ff60 0x7f86f0032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f87000096a0 tx=0x7f8700006b40).stop 2024-01-21T19:50:16.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.778+0000 7f870b9ba700 1 -- 172.21.15.173:0/3076100890 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87040f53f0 msgr2=0x7f87041055b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:16.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.778+0000 7f870b9ba700 1 --2- 172.21.15.173:0/3076100890 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87040f53f0 0x7f87041055b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f86f4013510 tx=0x7f86f400b040).stop 2024-01-21T19:50:16.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.778+0000 7f870b9ba700 1 -- 172.21.15.173:0/3076100890 shutdown_connections 2024-01-21T19:50:16.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.778+0000 7f870b9ba700 1 --2- 172.21.15.173:0/3076100890 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f86f002ff60 0x7f86f0032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:16.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.778+0000 7f870b9ba700 1 --2- 172.21.15.173:0/3076100890 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f87040f53f0 0x7f87041055b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:16.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.778+0000 7f870b9ba700 1 -- 172.21.15.173:0/3076100890 >> 172.21.15.173:0/3076100890 conn(0x7f87040f0b70 msgr2=0x7f87040f16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:16.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.778+0000 7f870b9ba700 1 -- 172.21.15.173:0/3076100890 shutdown_connections 2024-01-21T19:50:16.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:16.778+0000 7f870b9ba700 1 -- 172.21.15.173:0/3076100890 wait complete. 2024-01-21T19:50:16.772 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:50:17.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:16 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:14.947919+0000 mgr.smithi169.espgue (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:18.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:17 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:16.777396+0000 mon.smithi169 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.173:0/3076100890' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:50:18.242 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:50:18.242 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:50:19.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:18 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:16.948347+0000 mgr.smithi169.espgue (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:19.505 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.513+0000 7fda35241700 1 -- 172.21.15.173:0/3544981599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda30060280 msgr2=0x7fda300f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:19.505 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.513+0000 7fda35241700 1 --2- 172.21.15.173:0/3544981599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda30060280 0x7fda300f90f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fda18009fb0 tx=0x7fda18009b30).stop 2024-01-21T19:50:19.505 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.513+0000 7fda35241700 1 -- 172.21.15.173:0/3544981599 shutdown_connections 2024-01-21T19:50:19.506 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.513+0000 7fda35241700 1 --2- 172.21.15.173:0/3544981599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda30060280 0x7fda300f90f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:19.506 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.513+0000 7fda35241700 1 -- 172.21.15.173:0/3544981599 >> 172.21.15.173:0/3544981599 conn(0x7fda300f0b70 msgr2=0x7fda300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:19.506 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.513+0000 7fda35241700 1 -- 172.21.15.173:0/3544981599 shutdown_connections 2024-01-21T19:50:19.506 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.513+0000 7fda35241700 1 -- 172.21.15.173:0/3544981599 wait complete. 2024-01-21T19:50:19.507 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.514+0000 7fda35241700 1 Processor -- start 2024-01-21T19:50:19.507 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.514+0000 7fda35241700 1 -- start start 2024-01-21T19:50:19.507 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.514+0000 7fda35241700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda30060280 0x7fda301066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:19.507 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.514+0000 7fda35241700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda30106bd0 con 0x7fda30060280 2024-01-21T19:50:19.507 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.514+0000 7fda2ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda30060280 0x7fda301066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:19.508 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.514+0000 7fda2ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda30060280 0x7fda301066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:37636/0 (socket says 172.21.15.173:37636) 2024-01-21T19:50:19.508 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.514+0000 7fda2ed9d700 1 -- 172.21.15.173:0/473724626 learned_addr learned my addr 172.21.15.173:0/473724626 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:50:19.508 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.515+0000 7fda2ed9d700 1 -- 172.21.15.173:0/473724626 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda18005040 con 0x7fda30060280 2024-01-21T19:50:19.508 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.515+0000 7fda2ed9d700 1 --2- 172.21.15.173:0/473724626 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda30060280 0x7fda301066e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fda18013510 tx=0x7fda1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:19.508 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.515+0000 7fda277fe700 1 -- 172.21.15.173:0/473724626 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda18009320 con 0x7fda30060280 2024-01-21T19:50:19.508 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.515+0000 7fda35241700 1 -- 172.21.15.173:0/473724626 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda30106dd0 con 0x7fda30060280 2024-01-21T19:50:19.508 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.515+0000 7fda277fe700 1 -- 172.21.15.173:0/473724626 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda18013f00 con 0x7fda30060280 2024-01-21T19:50:19.508 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.515+0000 7fda277fe700 1 -- 172.21.15.173:0/473724626 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda18014dc0 con 0x7fda30060280 2024-01-21T19:50:19.508 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.515+0000 7fda35241700 1 -- 172.21.15.173:0/473724626 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda30107260 con 0x7fda30060280 2024-01-21T19:50:19.509 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.516+0000 7fda277fe700 1 -- 172.21.15.173:0/473724626 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fda180095f0 con 0x7fda30060280 2024-01-21T19:50:19.509 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.516+0000 7fda35241700 1 -- 172.21.15.173:0/473724626 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda30045ad0 con 0x7fda30060280 2024-01-21T19:50:19.509 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.516+0000 7fda277fe700 1 --2- 172.21.15.173:0/473724626 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fda1c02ff60 0x7fda1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:19.509 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.516+0000 7fda277fe700 1 -- 172.21.15.173:0/473724626 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fda18041040 con 0x7fda30060280 2024-01-21T19:50:19.509 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.516+0000 7fda2e59c700 1 --2- 172.21.15.173:0/473724626 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fda1c02ff60 0x7fda1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:19.509 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.516+0000 7fda2e59c700 1 --2- 172.21.15.173:0/473724626 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fda1c02ff60 0x7fda1c032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fda200096a0 tx=0x7fda20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:19.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.520+0000 7fda277fe700 1 -- 172.21.15.173:0/473724626 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda18014420 con 0x7fda30060280 2024-01-21T19:50:19.721 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.729+0000 7fda35241700 1 -- 172.21.15.173:0/473724626 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda301080a0 con 0x7fda30060280 2024-01-21T19:50:19.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.729+0000 7fda277fe700 1 -- 172.21.15.173:0/473724626 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fda18010230 con 0x7fda30060280 2024-01-21T19:50:19.722 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:50:19.722 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:50:19.723 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.731+0000 7fda35241700 1 -- 172.21.15.173:0/473724626 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fda1c02ff60 msgr2=0x7fda1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:19.723 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.731+0000 7fda35241700 1 --2- 172.21.15.173:0/473724626 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fda1c02ff60 0x7fda1c032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fda200096a0 tx=0x7fda20006b40).stop 2024-01-21T19:50:19.723 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.731+0000 7fda35241700 1 -- 172.21.15.173:0/473724626 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda30060280 msgr2=0x7fda301066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:19.724 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.731+0000 7fda35241700 1 --2- 172.21.15.173:0/473724626 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda30060280 0x7fda301066e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fda18013510 tx=0x7fda1800b040).stop 2024-01-21T19:50:19.724 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.731+0000 7fda35241700 1 -- 172.21.15.173:0/473724626 shutdown_connections 2024-01-21T19:50:19.724 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.731+0000 7fda35241700 1 --2- 172.21.15.173:0/473724626 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fda1c02ff60 0x7fda1c032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:19.724 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.731+0000 7fda35241700 1 --2- 172.21.15.173:0/473724626 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda30060280 0x7fda301066e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:19.724 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.731+0000 7fda35241700 1 -- 172.21.15.173:0/473724626 >> 172.21.15.173:0/473724626 conn(0x7fda300f0b70 msgr2=0x7fda300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:19.724 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.731+0000 7fda35241700 1 -- 172.21.15.173:0/473724626 shutdown_connections 2024-01-21T19:50:19.724 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:19.731+0000 7fda35241700 1 -- 172.21.15.173:0/473724626 wait complete. 2024-01-21T19:50:19.724 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:50:20.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:19 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:19.730131+0000 mon.smithi169 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.173:0/473724626' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:50:21.164 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:50:21.164 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:50:21.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:20 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:18.948770+0000 mgr.smithi169.espgue (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:22.394 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.401+0000 7f2dd550d700 1 -- 172.21.15.173:0/2219937466 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dd00f6510 msgr2=0x7f2dd00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:22.394 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.401+0000 7f2dd550d700 1 --2- 172.21.15.173:0/2219937466 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dd00f6510 0x7f2dd00f6930 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f2dc4009fb0 tx=0x7f2dc4009b30).stop 2024-01-21T19:50:22.394 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.401+0000 7f2dd550d700 1 -- 172.21.15.173:0/2219937466 shutdown_connections 2024-01-21T19:50:22.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.401+0000 7f2dd550d700 1 --2- 172.21.15.173:0/2219937466 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dd00f6510 0x7f2dd00f6930 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:22.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.401+0000 7f2dd550d700 1 -- 172.21.15.173:0/2219937466 >> 172.21.15.173:0/2219937466 conn(0x7f2dd00f1c70 msgr2=0x7f2dd00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:22.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.401+0000 7f2dd550d700 1 -- 172.21.15.173:0/2219937466 shutdown_connections 2024-01-21T19:50:22.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.401+0000 7f2dd550d700 1 -- 172.21.15.173:0/2219937466 wait complete. 2024-01-21T19:50:22.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.402+0000 7f2dd550d700 1 Processor -- start 2024-01-21T19:50:22.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.402+0000 7f2dd550d700 1 -- start start 2024-01-21T19:50:22.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.402+0000 7f2dd550d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dd00f6510 0x7f2dd0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:22.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.402+0000 7f2dd550d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd0106c20 con 0x7f2dd00f6510 2024-01-21T19:50:22.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.402+0000 7f2dceffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dd00f6510 0x7f2dd0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:22.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.402+0000 7f2dceffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dd00f6510 0x7f2dd0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:37656/0 (socket says 172.21.15.173:37656) 2024-01-21T19:50:22.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.402+0000 7f2dceffd700 1 -- 172.21.15.173:0/1137844249 learned_addr learned my addr 172.21.15.173:0/1137844249 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:50:22.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.403+0000 7f2dceffd700 1 -- 172.21.15.173:0/1137844249 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc4005040 con 0x7f2dd00f6510 2024-01-21T19:50:22.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.403+0000 7f2dceffd700 1 --2- 172.21.15.173:0/1137844249 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dd00f6510 0x7f2dd0106730 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f2dc4013510 tx=0x7f2dc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:22.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.403+0000 7f2db77fe700 1 -- 172.21.15.173:0/1137844249 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dc4009320 con 0x7f2dd00f6510 2024-01-21T19:50:22.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.403+0000 7f2dd550d700 1 -- 172.21.15.173:0/1137844249 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd0106e20 con 0x7f2dd00f6510 2024-01-21T19:50:22.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.403+0000 7f2db77fe700 1 -- 172.21.15.173:0/1137844249 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2dc4013f00 con 0x7f2dd00f6510 2024-01-21T19:50:22.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.403+0000 7f2dd550d700 1 -- 172.21.15.173:0/1137844249 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd01072b0 con 0x7f2dd00f6510 2024-01-21T19:50:22.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.403+0000 7f2db77fe700 1 -- 172.21.15.173:0/1137844249 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dc4014db0 con 0x7f2dd00f6510 2024-01-21T19:50:22.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.404+0000 7f2db77fe700 1 -- 172.21.15.173:0/1137844249 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2dc401e420 con 0x7f2dd00f6510 2024-01-21T19:50:22.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.404+0000 7f2dd550d700 1 -- 172.21.15.173:0/1137844249 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db0004fa0 con 0x7f2dd00f6510 2024-01-21T19:50:22.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.404+0000 7f2db77fe700 1 --2- 172.21.15.173:0/1137844249 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2db802ff60 0x7f2db8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:22.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.404+0000 7f2db77fe700 1 -- 172.21.15.173:0/1137844249 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2dc4040ee0 con 0x7f2dd00f6510 2024-01-21T19:50:22.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.404+0000 7f2dce7fc700 1 --2- 172.21.15.173:0/1137844249 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2db802ff60 0x7f2db8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:22.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.404+0000 7f2dce7fc700 1 --2- 172.21.15.173:0/1137844249 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2db802ff60 0x7f2db8032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f2dc00096a0 tx=0x7f2dc0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:22.401 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.409+0000 7f2db77fe700 1 -- 172.21.15.173:0/1137844249 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2dc401cd50 con 0x7f2dd00f6510 2024-01-21T19:50:22.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.611+0000 7f2dd550d700 1 -- 172.21.15.173:0/1137844249 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2db0005670 con 0x7f2dd00f6510 2024-01-21T19:50:22.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.612+0000 7f2db77fe700 1 -- 172.21.15.173:0/1137844249 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2dc402b2d0 con 0x7f2dd00f6510 2024-01-21T19:50:22.605 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:50:22.605 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:50:22.606 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.614+0000 7f2dd550d700 1 -- 172.21.15.173:0/1137844249 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2db802ff60 msgr2=0x7f2db8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:22.606 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.614+0000 7f2dd550d700 1 --2- 172.21.15.173:0/1137844249 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2db802ff60 0x7f2db8032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f2dc00096a0 tx=0x7f2dc0006b40).stop 2024-01-21T19:50:22.606 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.614+0000 7f2dd550d700 1 -- 172.21.15.173:0/1137844249 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dd00f6510 msgr2=0x7f2dd0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:22.606 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.614+0000 7f2dd550d700 1 --2- 172.21.15.173:0/1137844249 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dd00f6510 0x7f2dd0106730 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f2dc4013510 tx=0x7f2dc400b040).stop 2024-01-21T19:50:22.606 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.614+0000 7f2dd550d700 1 -- 172.21.15.173:0/1137844249 shutdown_connections 2024-01-21T19:50:22.606 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.614+0000 7f2dd550d700 1 --2- 172.21.15.173:0/1137844249 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2db802ff60 0x7f2db8032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:22.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.614+0000 7f2dd550d700 1 --2- 172.21.15.173:0/1137844249 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2dd00f6510 0x7f2dd0106730 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:22.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.614+0000 7f2dd550d700 1 -- 172.21.15.173:0/1137844249 >> 172.21.15.173:0/1137844249 conn(0x7f2dd00f1c70 msgr2=0x7f2dd00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:22.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.614+0000 7f2dd550d700 1 -- 172.21.15.173:0/1137844249 shutdown_connections 2024-01-21T19:50:22.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:22.614+0000 7f2dd550d700 1 -- 172.21.15.173:0/1137844249 wait complete. 2024-01-21T19:50:22.607 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:50:23.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:22 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:20.949109+0000 mgr.smithi169.espgue (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:23.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:22 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:22.612690+0000 mon.smithi169 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.173:0/1137844249' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:50:24.035 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:50:24.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:50:25.167 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:24 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:22.949412+0000 mgr.smithi169.espgue (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:25.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.295+0000 7f56d45e5700 1 -- 172.21.15.173:0/1483075775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56cc0f5410 msgr2=0x7f56cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:25.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.295+0000 7f56d45e5700 1 --2- 172.21.15.173:0/1483075775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56cc0f5410 0x7f56cc0f5830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f56bc009fb0 tx=0x7f56bc009b30).stop 2024-01-21T19:50:25.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.295+0000 7f56d45e5700 1 -- 172.21.15.173:0/1483075775 shutdown_connections 2024-01-21T19:50:25.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.295+0000 7f56d45e5700 1 --2- 172.21.15.173:0/1483075775 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56cc0f5410 0x7f56cc0f5830 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:25.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.295+0000 7f56d45e5700 1 -- 172.21.15.173:0/1483075775 >> 172.21.15.173:0/1483075775 conn(0x7f56cc0f0b70 msgr2=0x7f56cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:25.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.295+0000 7f56d45e5700 1 -- 172.21.15.173:0/1483075775 shutdown_connections 2024-01-21T19:50:25.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.295+0000 7f56d45e5700 1 -- 172.21.15.173:0/1483075775 wait complete. 2024-01-21T19:50:25.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.296+0000 7f56d45e5700 1 Processor -- start 2024-01-21T19:50:25.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.296+0000 7f56d45e5700 1 -- start start 2024-01-21T19:50:25.290 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.296+0000 7f56d45e5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56cc0f5410 0x7f56cc109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:25.290 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.296+0000 7f56d45e5700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc060220 con 0x7f56cc0f5410 2024-01-21T19:50:25.290 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.296+0000 7f56d2381700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56cc0f5410 0x7f56cc109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:25.290 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.296+0000 7f56d2381700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56cc0f5410 0x7f56cc109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:56916/0 (socket says 172.21.15.173:56916) 2024-01-21T19:50:25.290 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.296+0000 7f56d2381700 1 -- 172.21.15.173:0/2785958369 learned_addr learned my addr 172.21.15.173:0/2785958369 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:50:25.290 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.296+0000 7f56d2381700 1 -- 172.21.15.173:0/2785958369 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56bc005040 con 0x7f56cc0f5410 2024-01-21T19:50:25.290 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.296+0000 7f56d2381700 1 --2- 172.21.15.173:0/2785958369 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56cc0f5410 0x7f56cc109450 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f56bc013510 tx=0x7f56bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:25.290 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.296+0000 7f56c2ffd700 1 -- 172.21.15.173:0/2785958369 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56bc009320 con 0x7f56cc0f5410 2024-01-21T19:50:25.290 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.297+0000 7f56d45e5700 1 -- 172.21.15.173:0/2785958369 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56cc060420 con 0x7f56cc0f5410 2024-01-21T19:50:25.290 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.297+0000 7f56c2ffd700 1 -- 172.21.15.173:0/2785958369 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f56bc013f00 con 0x7f56cc0f5410 2024-01-21T19:50:25.290 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.297+0000 7f56c2ffd700 1 -- 172.21.15.173:0/2785958369 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56bc014d90 con 0x7f56cc0f5410 2024-01-21T19:50:25.290 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.297+0000 7f56d45e5700 1 -- 172.21.15.173:0/2785958369 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56cc109b00 con 0x7f56cc0f5410 2024-01-21T19:50:25.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.297+0000 7f56c2ffd700 1 -- 172.21.15.173:0/2785958369 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f56bc0095f0 con 0x7f56cc0f5410 2024-01-21T19:50:25.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.297+0000 7f56d45e5700 1 -- 172.21.15.173:0/2785958369 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56cc059100 con 0x7f56cc0f5410 2024-01-21T19:50:25.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.297+0000 7f56c2ffd700 1 --2- 172.21.15.173:0/2785958369 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f56b802ff60 0x7f56b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:25.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.298+0000 7f56c2ffd700 1 -- 172.21.15.173:0/2785958369 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f56bc041000 con 0x7f56cc0f5410 2024-01-21T19:50:25.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.298+0000 7f56d1b80700 1 --2- 172.21.15.173:0/2785958369 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f56b802ff60 0x7f56b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:25.292 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.298+0000 7f56d1b80700 1 --2- 172.21.15.173:0/2785958369 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f56b802ff60 0x7f56b8032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f56c80096a0 tx=0x7f56c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:25.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.302+0000 7f56c2ffd700 1 -- 172.21.15.173:0/2785958369 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f56bc014420 con 0x7f56cc0f5410 2024-01-21T19:50:25.500 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.507+0000 7f56d45e5700 1 -- 172.21.15.173:0/2785958369 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f56cc045ad0 con 0x7f56cc0f5410 2024-01-21T19:50:25.500 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.508+0000 7f56c2ffd700 1 -- 172.21.15.173:0/2785958369 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f56bc010230 con 0x7f56cc0f5410 2024-01-21T19:50:25.501 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:50:25.501 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:50:25.502 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.510+0000 7f56d45e5700 1 -- 172.21.15.173:0/2785958369 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f56b802ff60 msgr2=0x7f56b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:25.502 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.510+0000 7f56d45e5700 1 --2- 172.21.15.173:0/2785958369 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f56b802ff60 0x7f56b8032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f56c80096a0 tx=0x7f56c8006b40).stop 2024-01-21T19:50:25.502 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.510+0000 7f56d45e5700 1 -- 172.21.15.173:0/2785958369 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56cc0f5410 msgr2=0x7f56cc109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:25.502 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.510+0000 7f56d45e5700 1 --2- 172.21.15.173:0/2785958369 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56cc0f5410 0x7f56cc109450 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f56bc013510 tx=0x7f56bc00b040).stop 2024-01-21T19:50:25.502 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.510+0000 7f56d45e5700 1 -- 172.21.15.173:0/2785958369 shutdown_connections 2024-01-21T19:50:25.503 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.510+0000 7f56d45e5700 1 --2- 172.21.15.173:0/2785958369 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f56b802ff60 0x7f56b8032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:25.503 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.510+0000 7f56d45e5700 1 --2- 172.21.15.173:0/2785958369 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56cc0f5410 0x7f56cc109450 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:25.503 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.510+0000 7f56d45e5700 1 -- 172.21.15.173:0/2785958369 >> 172.21.15.173:0/2785958369 conn(0x7f56cc0f0b70 msgr2=0x7f56cc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:25.503 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.510+0000 7f56d45e5700 1 -- 172.21.15.173:0/2785958369 shutdown_connections 2024-01-21T19:50:25.503 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:25.510+0000 7f56d45e5700 1 -- 172.21.15.173:0/2785958369 wait complete. 2024-01-21T19:50:25.503 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:50:26.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:25 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:25.508853+0000 mon.smithi169 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.173:0/2785958369' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:50:26.934 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:50:26.935 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:50:27.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:26 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:24.949684+0000 mgr.smithi169.espgue (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:28.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.175+0000 7f16ef2f2700 1 -- 172.21.15.173:0/2951890864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16e80f6510 msgr2=0x7f16e80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:28.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.175+0000 7f16ef2f2700 1 --2- 172.21.15.173:0/2951890864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16e80f6510 0x7f16e80f6930 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f16dc009fb0 tx=0x7f16dc009b30).stop 2024-01-21T19:50:28.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.175+0000 7f16ef2f2700 1 -- 172.21.15.173:0/2951890864 shutdown_connections 2024-01-21T19:50:28.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.175+0000 7f16ef2f2700 1 --2- 172.21.15.173:0/2951890864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16e80f6510 0x7f16e80f6930 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:28.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.175+0000 7f16ef2f2700 1 -- 172.21.15.173:0/2951890864 >> 172.21.15.173:0/2951890864 conn(0x7f16e80f1c70 msgr2=0x7f16e80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:28.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.175+0000 7f16ef2f2700 1 -- 172.21.15.173:0/2951890864 shutdown_connections 2024-01-21T19:50:28.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.175+0000 7f16ef2f2700 1 -- 172.21.15.173:0/2951890864 wait complete. 2024-01-21T19:50:28.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.175+0000 7f16ef2f2700 1 Processor -- start 2024-01-21T19:50:28.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.175+0000 7f16ef2f2700 1 -- start start 2024-01-21T19:50:28.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.176+0000 7f16ef2f2700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16e80f6510 0x7f16e8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:28.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.176+0000 7f16ef2f2700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e8106c20 con 0x7f16e80f6510 2024-01-21T19:50:28.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.176+0000 7f16ed08e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16e80f6510 0x7f16e8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:28.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.176+0000 7f16ed08e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16e80f6510 0x7f16e8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:56944/0 (socket says 172.21.15.173:56944) 2024-01-21T19:50:28.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.176+0000 7f16ed08e700 1 -- 172.21.15.173:0/2384735588 learned_addr learned my addr 172.21.15.173:0/2384735588 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:50:28.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.176+0000 7f16ed08e700 1 -- 172.21.15.173:0/2384735588 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16dc005040 con 0x7f16e80f6510 2024-01-21T19:50:28.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.176+0000 7f16ed08e700 1 --2- 172.21.15.173:0/2384735588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16e80f6510 0x7f16e8106730 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f16dc013510 tx=0x7f16dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:28.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.176+0000 7f16d9ffb700 1 -- 172.21.15.173:0/2384735588 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16dc009320 con 0x7f16e80f6510 2024-01-21T19:50:28.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.176+0000 7f16ef2f2700 1 -- 172.21.15.173:0/2384735588 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16e8106e20 con 0x7f16e80f6510 2024-01-21T19:50:28.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.177+0000 7f16d9ffb700 1 -- 172.21.15.173:0/2384735588 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16dc013f00 con 0x7f16e80f6510 2024-01-21T19:50:28.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.177+0000 7f16d9ffb700 1 -- 172.21.15.173:0/2384735588 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16dc014d90 con 0x7f16e80f6510 2024-01-21T19:50:28.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.177+0000 7f16ef2f2700 1 -- 172.21.15.173:0/2384735588 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16e81072b0 con 0x7f16e80f6510 2024-01-21T19:50:28.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.177+0000 7f16d9ffb700 1 -- 172.21.15.173:0/2384735588 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f16dc014420 con 0x7f16e80f6510 2024-01-21T19:50:28.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.177+0000 7f16ef2f2700 1 -- 172.21.15.173:0/2384735588 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16e8045ad0 con 0x7f16e80f6510 2024-01-21T19:50:28.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.178+0000 7f16d9ffb700 1 --2- 172.21.15.173:0/2384735588 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f16d402ff60 0x7f16d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:28.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.178+0000 7f16d9ffb700 1 -- 172.21.15.173:0/2384735588 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f16dc041000 con 0x7f16e80f6510 2024-01-21T19:50:28.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.178+0000 7f16ec88d700 1 --2- 172.21.15.173:0/2384735588 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f16d402ff60 0x7f16d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:28.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.178+0000 7f16ec88d700 1 --2- 172.21.15.173:0/2384735588 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f16d402ff60 0x7f16d4032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f16e40096a0 tx=0x7f16e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:28.175 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.182+0000 7f16d9ffb700 1 -- 172.21.15.173:0/2384735588 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16dc0095f0 con 0x7f16e80f6510 2024-01-21T19:50:28.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.385+0000 7f16ef2f2700 1 -- 172.21.15.173:0/2384735588 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16e80fbf60 con 0x7f16e80f6510 2024-01-21T19:50:28.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.385+0000 7f16d9ffb700 1 -- 172.21.15.173:0/2384735588 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f16dc01db70 con 0x7f16e80f6510 2024-01-21T19:50:28.378 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:50:28.378 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:50:28.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.387+0000 7f16ef2f2700 1 -- 172.21.15.173:0/2384735588 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f16d402ff60 msgr2=0x7f16d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:28.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.387+0000 7f16ef2f2700 1 --2- 172.21.15.173:0/2384735588 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f16d402ff60 0x7f16d4032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f16e40096a0 tx=0x7f16e4006b40).stop 2024-01-21T19:50:28.380 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.387+0000 7f16ef2f2700 1 -- 172.21.15.173:0/2384735588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16e80f6510 msgr2=0x7f16e8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:28.380 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.387+0000 7f16ef2f2700 1 --2- 172.21.15.173:0/2384735588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16e80f6510 0x7f16e8106730 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f16dc013510 tx=0x7f16dc00b040).stop 2024-01-21T19:50:28.380 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.387+0000 7f16ef2f2700 1 -- 172.21.15.173:0/2384735588 shutdown_connections 2024-01-21T19:50:28.380 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.387+0000 7f16ef2f2700 1 --2- 172.21.15.173:0/2384735588 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f16d402ff60 0x7f16d4032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:28.380 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.387+0000 7f16ef2f2700 1 --2- 172.21.15.173:0/2384735588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16e80f6510 0x7f16e8106730 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:28.380 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.387+0000 7f16ef2f2700 1 -- 172.21.15.173:0/2384735588 >> 172.21.15.173:0/2384735588 conn(0x7f16e80f1c70 msgr2=0x7f16e80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:28.380 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.387+0000 7f16ef2f2700 1 -- 172.21.15.173:0/2384735588 shutdown_connections 2024-01-21T19:50:28.380 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:28.387+0000 7f16ef2f2700 1 -- 172.21.15.173:0/2384735588 wait complete. 2024-01-21T19:50:28.380 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:50:29.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:28 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:26.949914+0000 mgr.smithi169.espgue (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:29.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:28 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:28.386012+0000 mon.smithi169 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.173:0/2384735588' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:50:29.772 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:50:29.773 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:50:30.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.997+0000 7f72aac69700 1 -- 172.21.15.173:0/429864025 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f72a4060280 msgr2=0x7f72a40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:30.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.997+0000 7f72aac69700 1 --2- 172.21.15.173:0/429864025 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f72a4060280 0x7f72a40f90f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f7298009fb0 tx=0x7f7298009b30).stop 2024-01-21T19:50:30.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.997+0000 7f72aac69700 1 -- 172.21.15.173:0/429864025 shutdown_connections 2024-01-21T19:50:30.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.997+0000 7f72aac69700 1 --2- 172.21.15.173:0/429864025 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f72a4060280 0x7f72a40f90f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:30.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.997+0000 7f72aac69700 1 -- 172.21.15.173:0/429864025 >> 172.21.15.173:0/429864025 conn(0x7f72a40f0b70 msgr2=0x7f72a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:30.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.997+0000 7f72aac69700 1 -- 172.21.15.173:0/429864025 shutdown_connections 2024-01-21T19:50:30.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.997+0000 7f72aac69700 1 -- 172.21.15.173:0/429864025 wait complete. 2024-01-21T19:50:30.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.997+0000 7f72aac69700 1 Processor -- start 2024-01-21T19:50:30.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.997+0000 7f72aac69700 1 -- start start 2024-01-21T19:50:30.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.998+0000 7f72aac69700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f72a4060280 0x7f72a41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:30.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.998+0000 7f72aac69700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a4106bd0 con 0x7f72a4060280 2024-01-21T19:50:30.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.998+0000 7f72a8a05700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f72a4060280 0x7f72a41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:30.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.998+0000 7f72a8a05700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f72a4060280 0x7f72a41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:56952/0 (socket says 172.21.15.173:56952) 2024-01-21T19:50:30.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.998+0000 7f72a8a05700 1 -- 172.21.15.173:0/1666659893 learned_addr learned my addr 172.21.15.173:0/1666659893 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:50:30.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.998+0000 7f72a8a05700 1 -- 172.21.15.173:0/1666659893 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7298005040 con 0x7f72a4060280 2024-01-21T19:50:30.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.998+0000 7f72a8a05700 1 --2- 172.21.15.173:0/1666659893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f72a4060280 0x7f72a41066e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f7298013510 tx=0x7f729800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:30.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.999+0000 7f72a17fa700 1 -- 172.21.15.173:0/1666659893 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7298009320 con 0x7f72a4060280 2024-01-21T19:50:30.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.999+0000 7f72aac69700 1 -- 172.21.15.173:0/1666659893 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a4106dd0 con 0x7f72a4060280 2024-01-21T19:50:30.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.999+0000 7f72a17fa700 1 -- 172.21.15.173:0/1666659893 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7298013f00 con 0x7f72a4060280 2024-01-21T19:50:30.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.999+0000 7f72a17fa700 1 -- 172.21.15.173:0/1666659893 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7298014dc0 con 0x7f72a4060280 2024-01-21T19:50:30.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.999+0000 7f72aac69700 1 -- 172.21.15.173:0/1666659893 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a4107260 con 0x7f72a4060280 2024-01-21T19:50:30.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:30.999+0000 7f72a17fa700 1 -- 172.21.15.173:0/1666659893 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72980095f0 con 0x7f72a4060280 2024-01-21T19:50:30.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:31.000+0000 7f72a17fa700 1 --2- 172.21.15.173:0/1666659893 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f729402ff60 0x7f7294032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:30.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:31.000+0000 7f72a17fa700 1 -- 172.21.15.173:0/1666659893 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7298041020 con 0x7f72a4060280 2024-01-21T19:50:30.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:31.000+0000 7f72aac69700 1 -- 172.21.15.173:0/1666659893 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a4045ad0 con 0x7f72a4060280 2024-01-21T19:50:30.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:31.000+0000 7f72a3fff700 1 --2- 172.21.15.173:0/1666659893 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f729402ff60 0x7f7294032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:30.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:31.000+0000 7f72a3fff700 1 --2- 172.21.15.173:0/1666659893 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f729402ff60 0x7f7294032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f72900096a0 tx=0x7f7290006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:30.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:31.004+0000 7f72a17fa700 1 -- 172.21.15.173:0/1666659893 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7298014420 con 0x7f72a4060280 2024-01-21T19:50:31.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:30 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:28.950317+0000 mgr.smithi169.espgue (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:31.203 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:31.211+0000 7f72aac69700 1 -- 172.21.15.173:0/1666659893 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72a4107fa0 con 0x7f72a4060280 2024-01-21T19:50:31.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:31.212+0000 7f72a17fa700 1 -- 172.21.15.173:0/1666659893 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f729802b020 con 0x7f72a4060280 2024-01-21T19:50:31.204 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:50:31.204 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:50:31.206 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:31.213+0000 7f72aac69700 1 -- 172.21.15.173:0/1666659893 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f729402ff60 msgr2=0x7f7294032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:31.206 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:31.213+0000 7f72aac69700 1 --2- 172.21.15.173:0/1666659893 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f729402ff60 0x7f7294032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f72900096a0 tx=0x7f7290006b40).stop 2024-01-21T19:50:31.206 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:31.213+0000 7f72aac69700 1 -- 172.21.15.173:0/1666659893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f72a4060280 msgr2=0x7f72a41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:31.206 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:31.213+0000 7f72aac69700 1 --2- 172.21.15.173:0/1666659893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f72a4060280 0x7f72a41066e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f7298013510 tx=0x7f729800b040).stop 2024-01-21T19:50:31.206 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:31.213+0000 7f72aac69700 1 -- 172.21.15.173:0/1666659893 shutdown_connections 2024-01-21T19:50:31.206 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:31.213+0000 7f72aac69700 1 --2- 172.21.15.173:0/1666659893 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f729402ff60 0x7f7294032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:31.206 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:31.213+0000 7f72aac69700 1 --2- 172.21.15.173:0/1666659893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f72a4060280 0x7f72a41066e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:31.206 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:31.213+0000 7f72aac69700 1 -- 172.21.15.173:0/1666659893 >> 172.21.15.173:0/1666659893 conn(0x7f72a40f0b70 msgr2=0x7f72a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:31.206 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:31.213+0000 7f72aac69700 1 -- 172.21.15.173:0/1666659893 shutdown_connections 2024-01-21T19:50:31.206 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:31.213+0000 7f72aac69700 1 -- 172.21.15.173:0/1666659893 wait complete. 2024-01-21T19:50:31.207 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:50:32.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:31.070684+0000 mon.smithi169 (mon.0) 151 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/mirror_snapshot_schedule"}]: dispatch 2024-01-21T19:50:32.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:31.074447+0000 mon.smithi169 (mon.0) 152 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/trash_purge_schedule"}]: dispatch 2024-01-21T19:50:32.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:31.212259+0000 mon.smithi169 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.173:0/1666659893' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:50:32.605 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:50:32.605 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:50:33.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:32 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:30.950653+0000 mgr.smithi169.espgue (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:33.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.842+0000 7f343ea0a700 1 -- 172.21.15.173:0/3701752208 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34380f5410 msgr2=0x7f34380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:33.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.842+0000 7f343ea0a700 1 --2- 172.21.15.173:0/3701752208 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34380f5410 0x7f34380f5830 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f3428009fb0 tx=0x7f3428009b30).stop 2024-01-21T19:50:33.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.842+0000 7f343ea0a700 1 -- 172.21.15.173:0/3701752208 shutdown_connections 2024-01-21T19:50:33.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.842+0000 7f343ea0a700 1 --2- 172.21.15.173:0/3701752208 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34380f5410 0x7f34380f5830 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:33.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.842+0000 7f343ea0a700 1 -- 172.21.15.173:0/3701752208 >> 172.21.15.173:0/3701752208 conn(0x7f34380f0b70 msgr2=0x7f34380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:33.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.843+0000 7f343ea0a700 1 -- 172.21.15.173:0/3701752208 shutdown_connections 2024-01-21T19:50:33.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.843+0000 7f343ea0a700 1 -- 172.21.15.173:0/3701752208 wait complete. 2024-01-21T19:50:33.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.843+0000 7f343ea0a700 1 Processor -- start 2024-01-21T19:50:33.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.843+0000 7f343ea0a700 1 -- start start 2024-01-21T19:50:33.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.843+0000 7f343ea0a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34380f5410 0x7f3438104520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:33.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.843+0000 7f343ea0a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3438104a10 con 0x7f34380f5410 2024-01-21T19:50:33.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.844+0000 7f3437fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34380f5410 0x7f3438104520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:33.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.844+0000 7f3437fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34380f5410 0x7f3438104520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:54816/0 (socket says 172.21.15.173:54816) 2024-01-21T19:50:33.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.844+0000 7f3437fff700 1 -- 172.21.15.173:0/1695641000 learned_addr learned my addr 172.21.15.173:0/1695641000 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:50:33.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.844+0000 7f3437fff700 1 -- 172.21.15.173:0/1695641000 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3428005040 con 0x7f34380f5410 2024-01-21T19:50:33.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.844+0000 7f3437fff700 1 --2- 172.21.15.173:0/1695641000 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34380f5410 0x7f3438104520 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f3428013510 tx=0x7f342800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:33.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.844+0000 7f3434ff9700 1 -- 172.21.15.173:0/1695641000 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3428009320 con 0x7f34380f5410 2024-01-21T19:50:33.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.844+0000 7f343ea0a700 1 -- 172.21.15.173:0/1695641000 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3438104c10 con 0x7f34380f5410 2024-01-21T19:50:33.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.844+0000 7f3434ff9700 1 -- 172.21.15.173:0/1695641000 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3428013f00 con 0x7f34380f5410 2024-01-21T19:50:33.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.844+0000 7f3434ff9700 1 -- 172.21.15.173:0/1695641000 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3428014d90 con 0x7f34380f5410 2024-01-21T19:50:33.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.844+0000 7f343ea0a700 1 -- 172.21.15.173:0/1695641000 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34381050a0 con 0x7f34380f5410 2024-01-21T19:50:33.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.845+0000 7f3434ff9700 1 -- 172.21.15.173:0/1695641000 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34280095f0 con 0x7f34380f5410 2024-01-21T19:50:33.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.845+0000 7f343ea0a700 1 -- 172.21.15.173:0/1695641000 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3438045ad0 con 0x7f34380f5410 2024-01-21T19:50:33.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.845+0000 7f3434ff9700 1 --2- 172.21.15.173:0/1695641000 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f34200321b0 0x7f3420034670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:33.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.845+0000 7f3434ff9700 1 -- 172.21.15.173:0/1695641000 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3428040fb0 con 0x7f34380f5410 2024-01-21T19:50:33.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.846+0000 7f34377fe700 1 --2- 172.21.15.173:0/1695641000 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f34200321b0 0x7f3420034670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:33.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.846+0000 7f34377fe700 1 --2- 172.21.15.173:0/1695641000 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f34200321b0 0x7f3420034670 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f342c0096a0 tx=0x7f342c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:33.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:33.850+0000 7f3434ff9700 1 -- 172.21.15.173:0/1695641000 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3428014420 con 0x7f34380f5410 2024-01-21T19:50:34.047 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:34.055+0000 7f343ea0a700 1 -- 172.21.15.173:0/1695641000 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34380f9d80 con 0x7f34380f5410 2024-01-21T19:50:34.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:34.056+0000 7f3434ff9700 1 -- 172.21.15.173:0/1695641000 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3428010230 con 0x7f34380f5410 2024-01-21T19:50:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:50:34.049 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:50:34.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:34.057+0000 7f343ea0a700 1 -- 172.21.15.173:0/1695641000 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f34200321b0 msgr2=0x7f3420034670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:34.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:34.057+0000 7f343ea0a700 1 --2- 172.21.15.173:0/1695641000 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f34200321b0 0x7f3420034670 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f342c0096a0 tx=0x7f342c006b40).stop 2024-01-21T19:50:34.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:34.057+0000 7f343ea0a700 1 -- 172.21.15.173:0/1695641000 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34380f5410 msgr2=0x7f3438104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:34.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:34.057+0000 7f343ea0a700 1 --2- 172.21.15.173:0/1695641000 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34380f5410 0x7f3438104520 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f3428013510 tx=0x7f342800b040).stop 2024-01-21T19:50:34.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:34.057+0000 7f343ea0a700 1 -- 172.21.15.173:0/1695641000 shutdown_connections 2024-01-21T19:50:34.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:34.057+0000 7f343ea0a700 1 --2- 172.21.15.173:0/1695641000 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f34200321b0 0x7f3420034670 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:34.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:34.057+0000 7f343ea0a700 1 --2- 172.21.15.173:0/1695641000 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34380f5410 0x7f3438104520 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:34.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:34.058+0000 7f343ea0a700 1 -- 172.21.15.173:0/1695641000 >> 172.21.15.173:0/1695641000 conn(0x7f34380f0b70 msgr2=0x7f34380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:34.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:34.058+0000 7f343ea0a700 1 -- 172.21.15.173:0/1695641000 shutdown_connections 2024-01-21T19:50:34.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:34.058+0000 7f343ea0a700 1 -- 172.21.15.173:0/1695641000 wait complete. 2024-01-21T19:50:34.051 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:50:35.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:34 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:32.950968+0000 mgr.smithi169.espgue (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:35.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:34 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:34.056546+0000 mon.smithi169 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.173:0/1695641000' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:50:35.470 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:50:35.470 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:50:36.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.648+0000 7f79483df700 1 -- 172.21.15.173:0/244102339 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79400601e0 msgr2=0x7f79400f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:36.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.648+0000 7f79483df700 1 --2- 172.21.15.173:0/244102339 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79400601e0 0x7f79400f9190 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f793c009fb0 tx=0x7f793c009b30).stop 2024-01-21T19:50:36.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.648+0000 7f79483df700 1 -- 172.21.15.173:0/244102339 shutdown_connections 2024-01-21T19:50:36.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.648+0000 7f79483df700 1 --2- 172.21.15.173:0/244102339 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79400601e0 0x7f79400f9190 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:36.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.648+0000 7f79483df700 1 -- 172.21.15.173:0/244102339 >> 172.21.15.173:0/244102339 conn(0x7f79400f0b70 msgr2=0x7f79400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:36.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.648+0000 7f79483df700 1 -- 172.21.15.173:0/244102339 shutdown_connections 2024-01-21T19:50:36.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.648+0000 7f79483df700 1 -- 172.21.15.173:0/244102339 wait complete. 2024-01-21T19:50:36.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.649+0000 7f79483df700 1 Processor -- start 2024-01-21T19:50:36.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.649+0000 7f79483df700 1 -- start start 2024-01-21T19:50:36.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.649+0000 7f79483df700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79400601e0 0x7f79401066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:36.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.649+0000 7f79483df700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7940106bd0 con 0x7f79400601e0 2024-01-21T19:50:36.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.649+0000 7f794617b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79400601e0 0x7f79401066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:36.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.649+0000 7f794617b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79400601e0 0x7f79401066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:54834/0 (socket says 172.21.15.173:54834) 2024-01-21T19:50:36.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.649+0000 7f794617b700 1 -- 172.21.15.173:0/3716498802 learned_addr learned my addr 172.21.15.173:0/3716498802 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:50:36.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.650+0000 7f794617b700 1 -- 172.21.15.173:0/3716498802 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f793c005040 con 0x7f79400601e0 2024-01-21T19:50:36.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.650+0000 7f794617b700 1 --2- 172.21.15.173:0/3716498802 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79400601e0 0x7f79401066e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f793c004dd0 tx=0x7f793c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:36.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.650+0000 7f7936ffd700 1 -- 172.21.15.173:0/3716498802 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f793c018070 con 0x7f79400601e0 2024-01-21T19:50:36.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.650+0000 7f7936ffd700 1 -- 172.21.15.173:0/3716498802 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f793c013410 con 0x7f79400601e0 2024-01-21T19:50:36.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.650+0000 7f79483df700 1 -- 172.21.15.173:0/3716498802 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7940106dd0 con 0x7f79400601e0 2024-01-21T19:50:36.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.650+0000 7f7936ffd700 1 -- 172.21.15.173:0/3716498802 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f793c01c3c0 con 0x7f79400601e0 2024-01-21T19:50:36.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.650+0000 7f79483df700 1 -- 172.21.15.173:0/3716498802 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7940107260 con 0x7f79400601e0 2024-01-21T19:50:36.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.651+0000 7f7936ffd700 1 -- 172.21.15.173:0/3716498802 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f793c01d420 con 0x7f79400601e0 2024-01-21T19:50:36.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.651+0000 7f79483df700 1 -- 172.21.15.173:0/3716498802 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7940045ad0 con 0x7f79400601e0 2024-01-21T19:50:36.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.651+0000 7f7936ffd700 1 --2- 172.21.15.173:0/3716498802 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f792c02ff60 0x7f792c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:36.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.651+0000 7f7936ffd700 1 -- 172.21.15.173:0/3716498802 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f793c041330 con 0x7f79400601e0 2024-01-21T19:50:36.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.651+0000 7f794597a700 1 --2- 172.21.15.173:0/3716498802 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f792c02ff60 0x7f792c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:36.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.652+0000 7f794597a700 1 --2- 172.21.15.173:0/3716498802 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f792c02ff60 0x7f792c032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f79300096a0 tx=0x7f7930006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:36.648 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.656+0000 7f7936ffd700 1 -- 172.21.15.173:0/3716498802 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f793c01d820 con 0x7f79400601e0 2024-01-21T19:50:36.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.866+0000 7f79483df700 1 -- 172.21.15.173:0/3716498802 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7940108110 con 0x7f79400601e0 2024-01-21T19:50:36.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.866+0000 7f7936ffd700 1 -- 172.21.15.173:0/3716498802 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f793c01b370 con 0x7f79400601e0 2024-01-21T19:50:36.859 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:50:36.859 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:50:36.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.868+0000 7f79483df700 1 -- 172.21.15.173:0/3716498802 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f792c02ff60 msgr2=0x7f792c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:36.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.868+0000 7f79483df700 1 --2- 172.21.15.173:0/3716498802 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f792c02ff60 0x7f792c032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f79300096a0 tx=0x7f7930006b40).stop 2024-01-21T19:50:36.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.868+0000 7f79483df700 1 -- 172.21.15.173:0/3716498802 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79400601e0 msgr2=0x7f79401066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:36.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.868+0000 7f79483df700 1 --2- 172.21.15.173:0/3716498802 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79400601e0 0x7f79401066e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f793c004dd0 tx=0x7f793c009b30).stop 2024-01-21T19:50:36.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.868+0000 7f79483df700 1 -- 172.21.15.173:0/3716498802 shutdown_connections 2024-01-21T19:50:36.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.868+0000 7f79483df700 1 --2- 172.21.15.173:0/3716498802 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f792c02ff60 0x7f792c032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:36.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.868+0000 7f79483df700 1 --2- 172.21.15.173:0/3716498802 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f79400601e0 0x7f79401066e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:36.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.868+0000 7f79483df700 1 -- 172.21.15.173:0/3716498802 >> 172.21.15.173:0/3716498802 conn(0x7f79400f0b70 msgr2=0x7f79400f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:36.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.868+0000 7f79483df700 1 -- 172.21.15.173:0/3716498802 shutdown_connections 2024-01-21T19:50:36.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:36.868+0000 7f79483df700 1 -- 172.21.15.173:0/3716498802 wait complete. 2024-01-21T19:50:36.861 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:50:37.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:36 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:34.951345+0000 mgr.smithi169.espgue (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:38.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:37 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:36.866989+0000 mon.smithi169 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.173:0/3716498802' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:50:38.295 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:50:38.295 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:50:39.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:36.951794+0000 mgr.smithi169.espgue (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:39.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.558+0000 7fc5ff159700 1 -- 172.21.15.173:0/3148925338 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5f8060280 msgr2=0x7fc5f80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:39.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.558+0000 7fc5ff159700 1 --2- 172.21.15.173:0/3148925338 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5f8060280 0x7fc5f80f90f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fc5ec009fb0 tx=0x7fc5ec009b30).stop 2024-01-21T19:50:39.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.558+0000 7fc5ff159700 1 -- 172.21.15.173:0/3148925338 shutdown_connections 2024-01-21T19:50:39.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.558+0000 7fc5ff159700 1 --2- 172.21.15.173:0/3148925338 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5f8060280 0x7fc5f80f90f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:39.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.558+0000 7fc5ff159700 1 -- 172.21.15.173:0/3148925338 >> 172.21.15.173:0/3148925338 conn(0x7fc5f80f0b70 msgr2=0x7fc5f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:39.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.558+0000 7fc5ff159700 1 -- 172.21.15.173:0/3148925338 shutdown_connections 2024-01-21T19:50:39.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.558+0000 7fc5ff159700 1 -- 172.21.15.173:0/3148925338 wait complete. 2024-01-21T19:50:39.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.558+0000 7fc5ff159700 1 Processor -- start 2024-01-21T19:50:39.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.559+0000 7fc5ff159700 1 -- start start 2024-01-21T19:50:39.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.559+0000 7fc5ff159700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5f8060280 0x7fc5f81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:39.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.559+0000 7fc5ff159700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f8106bd0 con 0x7fc5f8060280 2024-01-21T19:50:39.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.559+0000 7fc5fcef5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5f8060280 0x7fc5f81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:39.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.559+0000 7fc5fcef5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5f8060280 0x7fc5f81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:54844/0 (socket says 172.21.15.173:54844) 2024-01-21T19:50:39.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.559+0000 7fc5fcef5700 1 -- 172.21.15.173:0/2263366110 learned_addr learned my addr 172.21.15.173:0/2263366110 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:50:39.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.559+0000 7fc5fcef5700 1 -- 172.21.15.173:0/2263366110 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec005040 con 0x7fc5f8060280 2024-01-21T19:50:39.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.559+0000 7fc5fcef5700 1 --2- 172.21.15.173:0/2263366110 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5f8060280 0x7fc5f81066e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fc5ec013510 tx=0x7fc5ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:39.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.559+0000 7fc5f57fa700 1 -- 172.21.15.173:0/2263366110 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5ec009320 con 0x7fc5f8060280 2024-01-21T19:50:39.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.560+0000 7fc5f57fa700 1 -- 172.21.15.173:0/2263366110 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5ec013f00 con 0x7fc5f8060280 2024-01-21T19:50:39.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.560+0000 7fc5ff159700 1 -- 172.21.15.173:0/2263366110 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f8106dd0 con 0x7fc5f8060280 2024-01-21T19:50:39.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.560+0000 7fc5f57fa700 1 -- 172.21.15.173:0/2263366110 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5ec014dc0 con 0x7fc5f8060280 2024-01-21T19:50:39.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.560+0000 7fc5ff159700 1 -- 172.21.15.173:0/2263366110 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f8107260 con 0x7fc5f8060280 2024-01-21T19:50:39.554 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.560+0000 7fc5f57fa700 1 -- 172.21.15.173:0/2263366110 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc5ec014420 con 0x7fc5f8060280 2024-01-21T19:50:39.554 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.560+0000 7fc5ff159700 1 -- 172.21.15.173:0/2263366110 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f8045ad0 con 0x7fc5f8060280 2024-01-21T19:50:39.554 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.560+0000 7fc5f57fa700 1 --2- 172.21.15.173:0/2263366110 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc5e802ff60 0x7fc5e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:39.554 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.561+0000 7fc5f57fa700 1 -- 172.21.15.173:0/2263366110 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc5ec041110 con 0x7fc5f8060280 2024-01-21T19:50:39.554 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.561+0000 7fc5f7fff700 1 --2- 172.21.15.173:0/2263366110 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc5e802ff60 0x7fc5e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:39.554 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.561+0000 7fc5f7fff700 1 --2- 172.21.15.173:0/2263366110 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc5e802ff60 0x7fc5e8032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fc5e40096a0 tx=0x7fc5e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:39.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.566+0000 7fc5f57fa700 1 -- 172.21.15.173:0/2263366110 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc5ec0095f0 con 0x7fc5f8060280 2024-01-21T19:50:39.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.771+0000 7fc5ff159700 1 -- 172.21.15.173:0/2263366110 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc5f8108110 con 0x7fc5f8060280 2024-01-21T19:50:39.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.772+0000 7fc5f57fa700 1 -- 172.21.15.173:0/2263366110 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc5ec01cb10 con 0x7fc5f8060280 2024-01-21T19:50:39.765 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:50:39.765 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:50:39.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.773+0000 7fc5ff159700 1 -- 172.21.15.173:0/2263366110 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc5e802ff60 msgr2=0x7fc5e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:39.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.773+0000 7fc5ff159700 1 --2- 172.21.15.173:0/2263366110 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc5e802ff60 0x7fc5e8032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fc5e40096a0 tx=0x7fc5e4006b40).stop 2024-01-21T19:50:39.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.773+0000 7fc5ff159700 1 -- 172.21.15.173:0/2263366110 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5f8060280 msgr2=0x7fc5f81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:39.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.773+0000 7fc5ff159700 1 --2- 172.21.15.173:0/2263366110 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5f8060280 0x7fc5f81066e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fc5ec013510 tx=0x7fc5ec00b040).stop 2024-01-21T19:50:39.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.773+0000 7fc5ff159700 1 -- 172.21.15.173:0/2263366110 shutdown_connections 2024-01-21T19:50:39.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.773+0000 7fc5ff159700 1 --2- 172.21.15.173:0/2263366110 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc5e802ff60 0x7fc5e8032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:39.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.773+0000 7fc5ff159700 1 --2- 172.21.15.173:0/2263366110 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5f8060280 0x7fc5f81066e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:39.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.773+0000 7fc5ff159700 1 -- 172.21.15.173:0/2263366110 >> 172.21.15.173:0/2263366110 conn(0x7fc5f80f0b70 msgr2=0x7fc5f80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:39.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.773+0000 7fc5ff159700 1 -- 172.21.15.173:0/2263366110 shutdown_connections 2024-01-21T19:50:39.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:39.773+0000 7fc5ff159700 1 -- 172.21.15.173:0/2263366110 wait complete. 2024-01-21T19:50:39.768 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:50:41.151 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:50:41.151 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:50:41.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:40 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:38.952186+0000 mgr.smithi169.espgue (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:41.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:40 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:39.773434+0000 mon.smithi169 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.173:0/2263366110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:50:42.376 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.381+0000 7fd0efb45700 1 -- 172.21.15.173:0/20933671 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0e8060280 msgr2=0x7fd0e80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:42.376 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.381+0000 7fd0efb45700 1 --2- 172.21.15.173:0/20933671 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0e8060280 0x7fd0e80f90f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fd0d8009fb0 tx=0x7fd0d8009b30).stop 2024-01-21T19:50:42.376 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.381+0000 7fd0efb45700 1 -- 172.21.15.173:0/20933671 shutdown_connections 2024-01-21T19:50:42.376 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.381+0000 7fd0efb45700 1 --2- 172.21.15.173:0/20933671 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0e8060280 0x7fd0e80f90f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:42.376 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.381+0000 7fd0efb45700 1 -- 172.21.15.173:0/20933671 >> 172.21.15.173:0/20933671 conn(0x7fd0e80f0b70 msgr2=0x7fd0e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:42.376 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.382+0000 7fd0efb45700 1 -- 172.21.15.173:0/20933671 shutdown_connections 2024-01-21T19:50:42.376 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.382+0000 7fd0efb45700 1 -- 172.21.15.173:0/20933671 wait complete. 2024-01-21T19:50:42.376 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.382+0000 7fd0efb45700 1 Processor -- start 2024-01-21T19:50:42.376 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.382+0000 7fd0efb45700 1 -- start start 2024-01-21T19:50:42.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.382+0000 7fd0efb45700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0e8060280 0x7fd0e81088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:42.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.382+0000 7fd0efb45700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e8108de0 con 0x7fd0e8060280 2024-01-21T19:50:42.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.382+0000 7fd0ed8e1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0e8060280 0x7fd0e81088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:42.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.382+0000 7fd0ed8e1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0e8060280 0x7fd0e81088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:54856/0 (socket says 172.21.15.173:54856) 2024-01-21T19:50:42.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.382+0000 7fd0ed8e1700 1 -- 172.21.15.173:0/3146223547 learned_addr learned my addr 172.21.15.173:0/3146223547 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:50:42.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.383+0000 7fd0ed8e1700 1 -- 172.21.15.173:0/3146223547 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d8005040 con 0x7fd0e8060280 2024-01-21T19:50:42.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.383+0000 7fd0ed8e1700 1 --2- 172.21.15.173:0/3146223547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0e8060280 0x7fd0e81088f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fd0d8013510 tx=0x7fd0d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:42.378 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.383+0000 7fd0de7fc700 1 -- 172.21.15.173:0/3146223547 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0d8009320 con 0x7fd0e8060280 2024-01-21T19:50:42.378 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.383+0000 7fd0de7fc700 1 -- 172.21.15.173:0/3146223547 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd0d8013f00 con 0x7fd0e8060280 2024-01-21T19:50:42.378 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.383+0000 7fd0efb45700 1 -- 172.21.15.173:0/3146223547 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e8108fe0 con 0x7fd0e8060280 2024-01-21T19:50:42.378 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.383+0000 7fd0de7fc700 1 -- 172.21.15.173:0/3146223547 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0d8014dc0 con 0x7fd0e8060280 2024-01-21T19:50:42.378 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.383+0000 7fd0efb45700 1 -- 172.21.15.173:0/3146223547 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e8109470 con 0x7fd0e8060280 2024-01-21T19:50:42.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.384+0000 7fd0de7fc700 1 -- 172.21.15.173:0/3146223547 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd0d8014420 con 0x7fd0e8060280 2024-01-21T19:50:42.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.384+0000 7fd0efb45700 1 -- 172.21.15.173:0/3146223547 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e8045ad0 con 0x7fd0e8060280 2024-01-21T19:50:42.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.384+0000 7fd0de7fc700 1 --2- 172.21.15.173:0/3146223547 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd0d402ff60 0x7fd0d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:42.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.384+0000 7fd0de7fc700 1 -- 172.21.15.173:0/3146223547 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd0d8041050 con 0x7fd0e8060280 2024-01-21T19:50:42.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.384+0000 7fd0ed0e0700 1 --2- 172.21.15.173:0/3146223547 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd0d402ff60 0x7fd0d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:42.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.385+0000 7fd0ed0e0700 1 --2- 172.21.15.173:0/3146223547 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd0d402ff60 0x7fd0d4032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fd0e40096a0 tx=0x7fd0e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:42.382 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.389+0000 7fd0de7fc700 1 -- 172.21.15.173:0/3146223547 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd0d80095f0 con 0x7fd0e8060280 2024-01-21T19:50:42.592 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.598+0000 7fd0efb45700 1 -- 172.21.15.173:0/3146223547 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0e80fbf50 con 0x7fd0e8060280 2024-01-21T19:50:42.592 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.599+0000 7fd0de7fc700 1 -- 172.21.15.173:0/3146223547 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd0d801cb10 con 0x7fd0e8060280 2024-01-21T19:50:42.593 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:50:42.593 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:50:42.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.601+0000 7fd0efb45700 1 -- 172.21.15.173:0/3146223547 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd0d402ff60 msgr2=0x7fd0d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:42.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.601+0000 7fd0efb45700 1 --2- 172.21.15.173:0/3146223547 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd0d402ff60 0x7fd0d4032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fd0e40096a0 tx=0x7fd0e4006b40).stop 2024-01-21T19:50:42.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.601+0000 7fd0efb45700 1 -- 172.21.15.173:0/3146223547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0e8060280 msgr2=0x7fd0e81088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:42.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.601+0000 7fd0efb45700 1 --2- 172.21.15.173:0/3146223547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0e8060280 0x7fd0e81088f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fd0d8013510 tx=0x7fd0d800b040).stop 2024-01-21T19:50:42.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.601+0000 7fd0efb45700 1 -- 172.21.15.173:0/3146223547 shutdown_connections 2024-01-21T19:50:42.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.601+0000 7fd0efb45700 1 --2- 172.21.15.173:0/3146223547 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd0d402ff60 0x7fd0d4032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:42.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.601+0000 7fd0efb45700 1 --2- 172.21.15.173:0/3146223547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd0e8060280 0x7fd0e81088f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:42.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.601+0000 7fd0efb45700 1 -- 172.21.15.173:0/3146223547 >> 172.21.15.173:0/3146223547 conn(0x7fd0e80f0b70 msgr2=0x7fd0e80f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:42.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.601+0000 7fd0efb45700 1 -- 172.21.15.173:0/3146223547 shutdown_connections 2024-01-21T19:50:42.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:42.601+0000 7fd0efb45700 1 -- 172.21.15.173:0/3146223547 wait complete. 2024-01-21T19:50:42.595 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:50:43.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:40.952628+0000 mgr.smithi169.espgue (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:43.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:42.600779+0000 mon.smithi169 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.173:0/3146223547' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:50:44.031 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:50:44.031 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:50:45.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:44 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:42.952996+0000 mgr.smithi169.espgue (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:45.252 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.258+0000 7f99b00a3700 1 -- 172.21.15.173:0/4136962433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99a80f5410 msgr2=0x7f99a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:45.252 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.258+0000 7f99b00a3700 1 --2- 172.21.15.173:0/4136962433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99a80f5410 0x7f99a80f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f99a4009fb0 tx=0x7f99a4009b30).stop 2024-01-21T19:50:45.252 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.258+0000 7f99b00a3700 1 -- 172.21.15.173:0/4136962433 shutdown_connections 2024-01-21T19:50:45.252 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.258+0000 7f99b00a3700 1 --2- 172.21.15.173:0/4136962433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99a80f5410 0x7f99a80f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:45.253 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.258+0000 7f99b00a3700 1 -- 172.21.15.173:0/4136962433 >> 172.21.15.173:0/4136962433 conn(0x7f99a80f0b70 msgr2=0x7f99a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:45.253 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.258+0000 7f99b00a3700 1 -- 172.21.15.173:0/4136962433 shutdown_connections 2024-01-21T19:50:45.253 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.258+0000 7f99b00a3700 1 -- 172.21.15.173:0/4136962433 wait complete. 2024-01-21T19:50:45.253 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.259+0000 7f99b00a3700 1 Processor -- start 2024-01-21T19:50:45.253 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.259+0000 7f99b00a3700 1 -- start start 2024-01-21T19:50:45.253 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.259+0000 7f99b00a3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99a80f5410 0x7f99a8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:45.253 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.259+0000 7f99b00a3700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a8060240 con 0x7f99a80f5410 2024-01-21T19:50:45.253 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.259+0000 7f99ade3f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99a80f5410 0x7f99a8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:45.253 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.259+0000 7f99ade3f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99a80f5410 0x7f99a8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:35086/0 (socket says 172.21.15.173:35086) 2024-01-21T19:50:45.253 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.259+0000 7f99ade3f700 1 -- 172.21.15.173:0/61095266 learned_addr learned my addr 172.21.15.173:0/61095266 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:50:45.255 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.260+0000 7f99ade3f700 1 -- 172.21.15.173:0/61095266 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a4005040 con 0x7f99a80f5410 2024-01-21T19:50:45.255 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.260+0000 7f99ade3f700 1 --2- 172.21.15.173:0/61095266 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99a80f5410 0x7f99a8109430 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f99a4013510 tx=0x7f99a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:45.255 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.260+0000 7f999a7fc700 1 -- 172.21.15.173:0/61095266 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99a4009320 con 0x7f99a80f5410 2024-01-21T19:50:45.255 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.260+0000 7f99b00a3700 1 -- 172.21.15.173:0/61095266 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a8060440 con 0x7f99a80f5410 2024-01-21T19:50:45.255 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.260+0000 7f999a7fc700 1 -- 172.21.15.173:0/61095266 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99a4013f00 con 0x7f99a80f5410 2024-01-21T19:50:45.255 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.260+0000 7f999a7fc700 1 -- 172.21.15.173:0/61095266 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99a4014d90 con 0x7f99a80f5410 2024-01-21T19:50:45.255 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.260+0000 7f99b00a3700 1 -- 172.21.15.173:0/61095266 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a8109ae0 con 0x7f99a80f5410 2024-01-21T19:50:45.255 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.261+0000 7f999a7fc700 1 -- 172.21.15.173:0/61095266 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f99a40095f0 con 0x7f99a80f5410 2024-01-21T19:50:45.255 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.261+0000 7f99b00a3700 1 -- 172.21.15.173:0/61095266 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99a8045ad0 con 0x7f99a80f5410 2024-01-21T19:50:45.255 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.261+0000 7f999a7fc700 1 --2- 172.21.15.173:0/61095266 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f999402ff60 0x7f9994032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:45.255 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.261+0000 7f999a7fc700 1 -- 172.21.15.173:0/61095266 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f99a4041000 con 0x7f99a80f5410 2024-01-21T19:50:45.255 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.261+0000 7f99ad63e700 1 --2- 172.21.15.173:0/61095266 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f999402ff60 0x7f9994032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:45.256 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.261+0000 7f99ad63e700 1 --2- 172.21.15.173:0/61095266 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f999402ff60 0x7f9994032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f999c0096a0 tx=0x7f999c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:45.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.266+0000 7f999a7fc700 1 -- 172.21.15.173:0/61095266 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f99a4014420 con 0x7f99a80f5410 2024-01-21T19:50:45.463 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.470+0000 7f99b00a3700 1 -- 172.21.15.173:0/61095266 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f99a80fae60 con 0x7f99a80f5410 2024-01-21T19:50:45.464 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.471+0000 7f999a7fc700 1 -- 172.21.15.173:0/61095266 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f99a4010230 con 0x7f99a80f5410 2024-01-21T19:50:45.464 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:50:45.464 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:50:45.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.472+0000 7f99b00a3700 1 -- 172.21.15.173:0/61095266 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f999402ff60 msgr2=0x7f9994032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:45.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.472+0000 7f99b00a3700 1 --2- 172.21.15.173:0/61095266 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f999402ff60 0x7f9994032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f999c0096a0 tx=0x7f999c006b40).stop 2024-01-21T19:50:45.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.472+0000 7f99b00a3700 1 -- 172.21.15.173:0/61095266 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99a80f5410 msgr2=0x7f99a8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:45.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.472+0000 7f99b00a3700 1 --2- 172.21.15.173:0/61095266 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99a80f5410 0x7f99a8109430 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f99a4013510 tx=0x7f99a400b040).stop 2024-01-21T19:50:45.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.472+0000 7f99b00a3700 1 -- 172.21.15.173:0/61095266 shutdown_connections 2024-01-21T19:50:45.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.472+0000 7f99b00a3700 1 --2- 172.21.15.173:0/61095266 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f999402ff60 0x7f9994032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:45.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.472+0000 7f99b00a3700 1 --2- 172.21.15.173:0/61095266 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99a80f5410 0x7f99a8109430 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:45.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.472+0000 7f99b00a3700 1 -- 172.21.15.173:0/61095266 >> 172.21.15.173:0/61095266 conn(0x7f99a80f0b70 msgr2=0x7f99a80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:45.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.472+0000 7f99b00a3700 1 -- 172.21.15.173:0/61095266 shutdown_connections 2024-01-21T19:50:45.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:45.472+0000 7f99b00a3700 1 -- 172.21.15.173:0/61095266 wait complete. 2024-01-21T19:50:45.467 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:50:46.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:45 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:45.472168+0000 mon.smithi169 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.173:0/61095266' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:50:46.879 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:50:46.880 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:50:47.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:46 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:44.953336+0000 mgr.smithi169.espgue (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:48.030 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.035+0000 7f473bfff700 1 -- 172.21.15.173:0/1217302232 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f473c0f51a0 msgr2=0x7f473c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:48.030 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.035+0000 7f473bfff700 1 --2- 172.21.15.173:0/1217302232 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f473c0f51a0 0x7f473c0f55c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f472c009fb0 tx=0x7f472c009b30).stop 2024-01-21T19:50:48.030 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.035+0000 7f473bfff700 1 -- 172.21.15.173:0/1217302232 shutdown_connections 2024-01-21T19:50:48.030 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.035+0000 7f473bfff700 1 --2- 172.21.15.173:0/1217302232 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f473c0f51a0 0x7f473c0f55c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:48.030 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.035+0000 7f473bfff700 1 -- 172.21.15.173:0/1217302232 >> 172.21.15.173:0/1217302232 conn(0x7f473c0f09a0 msgr2=0x7f473c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:48.030 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.035+0000 7f473bfff700 1 -- 172.21.15.173:0/1217302232 shutdown_connections 2024-01-21T19:50:48.030 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.035+0000 7f473bfff700 1 -- 172.21.15.173:0/1217302232 wait complete. 2024-01-21T19:50:48.030 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.035+0000 7f473bfff700 1 Processor -- start 2024-01-21T19:50:48.031 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.035+0000 7f473bfff700 1 -- start start 2024-01-21T19:50:48.031 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.036+0000 7f473bfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f473c0f51a0 0x7f473c1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:48.031 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.036+0000 7f473bfff700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473c1058a0 con 0x7f473c0f51a0 2024-01-21T19:50:48.031 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.036+0000 7f473affd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f473c0f51a0 0x7f473c1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:48.031 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.036+0000 7f473affd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f473c0f51a0 0x7f473c1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:35104/0 (socket says 172.21.15.173:35104) 2024-01-21T19:50:48.031 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.036+0000 7f473affd700 1 -- 172.21.15.173:0/371027243 learned_addr learned my addr 172.21.15.173:0/371027243 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:50:48.031 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.036+0000 7f473affd700 1 -- 172.21.15.173:0/371027243 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c005040 con 0x7f473c0f51a0 2024-01-21T19:50:48.031 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.036+0000 7f473affd700 1 --2- 172.21.15.173:0/371027243 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f473c0f51a0 0x7f473c1053b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f472c013510 tx=0x7f472c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:48.031 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.036+0000 7f47237fe700 1 -- 172.21.15.173:0/371027243 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f472c009320 con 0x7f473c0f51a0 2024-01-21T19:50:48.031 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.036+0000 7f473bfff700 1 -- 172.21.15.173:0/371027243 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f473c105aa0 con 0x7f473c0f51a0 2024-01-21T19:50:48.031 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.036+0000 7f47237fe700 1 -- 172.21.15.173:0/371027243 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f472c013f00 con 0x7f473c0f51a0 2024-01-21T19:50:48.031 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.036+0000 7f47237fe700 1 -- 172.21.15.173:0/371027243 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f472c014dc0 con 0x7f473c0f51a0 2024-01-21T19:50:48.031 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.036+0000 7f473bfff700 1 -- 172.21.15.173:0/371027243 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f473c105f30 con 0x7f473c0f51a0 2024-01-21T19:50:48.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.037+0000 7f47237fe700 1 -- 172.21.15.173:0/371027243 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f472c01d3d0 con 0x7f473c0f51a0 2024-01-21T19:50:48.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.037+0000 7f473bfff700 1 -- 172.21.15.173:0/371027243 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f473c045ad0 con 0x7f473c0f51a0 2024-01-21T19:50:48.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.037+0000 7f47237fe700 1 --2- 172.21.15.173:0/371027243 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f472402ff60 0x7f4724032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:48.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.037+0000 7f47237fe700 1 -- 172.21.15.173:0/371027243 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f472c0412f0 con 0x7f473c0f51a0 2024-01-21T19:50:48.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.038+0000 7f473a7fc700 1 --2- 172.21.15.173:0/371027243 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f472402ff60 0x7f4724032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:48.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.038+0000 7f473a7fc700 1 --2- 172.21.15.173:0/371027243 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f472402ff60 0x7f4724032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f47300096a0 tx=0x7f4730006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:48.035 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.042+0000 7f47237fe700 1 -- 172.21.15.173:0/371027243 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f472c01ccf0 con 0x7f473c0f51a0 2024-01-21T19:50:48.241 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.248+0000 7f473bfff700 1 -- 172.21.15.173:0/371027243 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f473c106de0 con 0x7f473c0f51a0 2024-01-21T19:50:48.242 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.249+0000 7f47237fe700 1 -- 172.21.15.173:0/371027243 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f472c01cb10 con 0x7f473c0f51a0 2024-01-21T19:50:48.242 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:50:48.242 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:50:48.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.250+0000 7f473bfff700 1 -- 172.21.15.173:0/371027243 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f472402ff60 msgr2=0x7f4724032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:48.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.250+0000 7f473bfff700 1 --2- 172.21.15.173:0/371027243 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f472402ff60 0x7f4724032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f47300096a0 tx=0x7f4730006b40).stop 2024-01-21T19:50:48.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.250+0000 7f473bfff700 1 -- 172.21.15.173:0/371027243 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f473c0f51a0 msgr2=0x7f473c1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:48.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.250+0000 7f473bfff700 1 --2- 172.21.15.173:0/371027243 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f473c0f51a0 0x7f473c1053b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f472c013510 tx=0x7f472c00b040).stop 2024-01-21T19:50:48.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.250+0000 7f473bfff700 1 -- 172.21.15.173:0/371027243 shutdown_connections 2024-01-21T19:50:48.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.250+0000 7f473bfff700 1 --2- 172.21.15.173:0/371027243 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f472402ff60 0x7f4724032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:48.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.250+0000 7f473bfff700 1 --2- 172.21.15.173:0/371027243 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f473c0f51a0 0x7f473c1053b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:48.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.250+0000 7f473bfff700 1 -- 172.21.15.173:0/371027243 >> 172.21.15.173:0/371027243 conn(0x7f473c0f09a0 msgr2=0x7f473c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:48.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.250+0000 7f473bfff700 1 -- 172.21.15.173:0/371027243 shutdown_connections 2024-01-21T19:50:48.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:48.250+0000 7f473bfff700 1 -- 172.21.15.173:0/371027243 wait complete. 2024-01-21T19:50:48.245 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:50:49.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:46.953709+0000 mgr.smithi169.espgue (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:49.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:48.250163+0000 mon.smithi169 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.173:0/371027243' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:50:49.631 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:50:49.631 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:50:50.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.894+0000 7f4e00eac700 1 -- 172.21.15.173:0/3455020777 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dfc0f6510 msgr2=0x7f4dfc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:50.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.894+0000 7f4e00eac700 1 --2- 172.21.15.173:0/3455020777 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dfc0f6510 0x7f4dfc0f6930 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f4df0009fb0 tx=0x7f4df0009b30).stop 2024-01-21T19:50:50.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.894+0000 7f4e00eac700 1 -- 172.21.15.173:0/3455020777 shutdown_connections 2024-01-21T19:50:50.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.894+0000 7f4e00eac700 1 --2- 172.21.15.173:0/3455020777 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dfc0f6510 0x7f4dfc0f6930 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:50.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.894+0000 7f4e00eac700 1 -- 172.21.15.173:0/3455020777 >> 172.21.15.173:0/3455020777 conn(0x7f4dfc0f1c70 msgr2=0x7f4dfc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:50.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.894+0000 7f4e00eac700 1 -- 172.21.15.173:0/3455020777 shutdown_connections 2024-01-21T19:50:50.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.894+0000 7f4e00eac700 1 -- 172.21.15.173:0/3455020777 wait complete. 2024-01-21T19:50:50.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.895+0000 7f4e00eac700 1 Processor -- start 2024-01-21T19:50:50.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.895+0000 7f4e00eac700 1 -- start start 2024-01-21T19:50:50.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.895+0000 7f4e00eac700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dfc0f6510 0x7f4dfc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:50.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.895+0000 7f4e00eac700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc106c20 con 0x7f4dfc0f6510 2024-01-21T19:50:50.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.895+0000 7f4dfa59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dfc0f6510 0x7f4dfc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:50.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.895+0000 7f4dfa59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dfc0f6510 0x7f4dfc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:35110/0 (socket says 172.21.15.173:35110) 2024-01-21T19:50:50.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.895+0000 7f4dfa59c700 1 -- 172.21.15.173:0/801436095 learned_addr learned my addr 172.21.15.173:0/801436095 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:50:50.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.895+0000 7f4dfa59c700 1 -- 172.21.15.173:0/801436095 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df0005040 con 0x7f4dfc0f6510 2024-01-21T19:50:50.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.896+0000 7f4dfa59c700 1 --2- 172.21.15.173:0/801436095 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dfc0f6510 0x7f4dfc106730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f4df0013510 tx=0x7f4df000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:50.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.896+0000 7f4deaffd700 1 -- 172.21.15.173:0/801436095 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4df0009320 con 0x7f4dfc0f6510 2024-01-21T19:50:50.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.896+0000 7f4e00eac700 1 -- 172.21.15.173:0/801436095 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dfc106e20 con 0x7f4dfc0f6510 2024-01-21T19:50:50.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.896+0000 7f4deaffd700 1 -- 172.21.15.173:0/801436095 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4df0013f00 con 0x7f4dfc0f6510 2024-01-21T19:50:50.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.896+0000 7f4deaffd700 1 -- 172.21.15.173:0/801436095 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4df0014d90 con 0x7f4dfc0f6510 2024-01-21T19:50:50.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.896+0000 7f4e00eac700 1 -- 172.21.15.173:0/801436095 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dfc1072b0 con 0x7f4dfc0f6510 2024-01-21T19:50:50.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.896+0000 7f4deaffd700 1 -- 172.21.15.173:0/801436095 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4df00263a0 con 0x7f4dfc0f6510 2024-01-21T19:50:50.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.896+0000 7f4e00eac700 1 -- 172.21.15.173:0/801436095 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dfc045ad0 con 0x7f4dfc0f6510 2024-01-21T19:50:50.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.897+0000 7f4deaffd700 1 --2- 172.21.15.173:0/801436095 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4de402ff60 0x7f4de4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:50.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.897+0000 7f4deaffd700 1 -- 172.21.15.173:0/801436095 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4df0041040 con 0x7f4dfc0f6510 2024-01-21T19:50:50.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.897+0000 7f4df9d9b700 1 --2- 172.21.15.173:0/801436095 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4de402ff60 0x7f4de4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:50.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.897+0000 7f4df9d9b700 1 --2- 172.21.15.173:0/801436095 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4de402ff60 0x7f4de4032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f4dec0096a0 tx=0x7f4dec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:50.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:50.901+0000 7f4deaffd700 1 -- 172.21.15.173:0/801436095 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4df00126f0 con 0x7f4dfc0f6510 2024-01-21T19:50:51.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:51.105+0000 7f4e00eac700 1 -- 172.21.15.173:0/801436095 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4dfc0fbf60 con 0x7f4dfc0f6510 2024-01-21T19:50:51.099 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:51.105+0000 7f4deaffd700 1 -- 172.21.15.173:0/801436095 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4df0012330 con 0x7f4dfc0f6510 2024-01-21T19:50:51.099 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:50:51.099 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:50:51.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:51.107+0000 7f4e00eac700 1 -- 172.21.15.173:0/801436095 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4de402ff60 msgr2=0x7f4de4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:51.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:51.107+0000 7f4e00eac700 1 --2- 172.21.15.173:0/801436095 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4de402ff60 0x7f4de4032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f4dec0096a0 tx=0x7f4dec006b40).stop 2024-01-21T19:50:51.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:51.107+0000 7f4e00eac700 1 -- 172.21.15.173:0/801436095 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dfc0f6510 msgr2=0x7f4dfc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:51.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:51.107+0000 7f4e00eac700 1 --2- 172.21.15.173:0/801436095 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dfc0f6510 0x7f4dfc106730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f4df0013510 tx=0x7f4df000b040).stop 2024-01-21T19:50:51.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:51.107+0000 7f4e00eac700 1 -- 172.21.15.173:0/801436095 shutdown_connections 2024-01-21T19:50:51.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:51.107+0000 7f4e00eac700 1 --2- 172.21.15.173:0/801436095 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4de402ff60 0x7f4de4032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:51.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:51.107+0000 7f4e00eac700 1 --2- 172.21.15.173:0/801436095 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dfc0f6510 0x7f4dfc106730 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:51.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:51.107+0000 7f4e00eac700 1 -- 172.21.15.173:0/801436095 >> 172.21.15.173:0/801436095 conn(0x7f4dfc0f1c70 msgr2=0x7f4dfc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:51.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:51.107+0000 7f4e00eac700 1 -- 172.21.15.173:0/801436095 shutdown_connections 2024-01-21T19:50:51.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:51.107+0000 7f4e00eac700 1 -- 172.21.15.173:0/801436095 wait complete. 2024-01-21T19:50:51.101 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:50:51.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:50 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:48.954069+0000 mgr.smithi169.espgue (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:52.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:51 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:51.107074+0000 mon.smithi169 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.173:0/801436095' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:50:52.487 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:50:52.487 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:50:53.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:52 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:50.954322+0000 mgr.smithi169.espgue (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:53.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.688+0000 7fd18d9fb700 1 -- 172.21.15.173:0/2593647523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1880f62c0 msgr2=0x7fd1880f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:53.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.688+0000 7fd18d9fb700 1 --2- 172.21.15.173:0/2593647523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1880f62c0 0x7fd1880f66e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fd17c009fb0 tx=0x7fd17c009b30).stop 2024-01-21T19:50:53.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.688+0000 7fd18d9fb700 1 -- 172.21.15.173:0/2593647523 shutdown_connections 2024-01-21T19:50:53.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.688+0000 7fd18d9fb700 1 --2- 172.21.15.173:0/2593647523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1880f62c0 0x7fd1880f66e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:53.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.688+0000 7fd18d9fb700 1 -- 172.21.15.173:0/2593647523 >> 172.21.15.173:0/2593647523 conn(0x7fd1880f1ac0 msgr2=0x7fd1880f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:53.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.688+0000 7fd18d9fb700 1 -- 172.21.15.173:0/2593647523 shutdown_connections 2024-01-21T19:50:53.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.688+0000 7fd18d9fb700 1 -- 172.21.15.173:0/2593647523 wait complete. 2024-01-21T19:50:53.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.688+0000 7fd18d9fb700 1 Processor -- start 2024-01-21T19:50:53.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.688+0000 7fd18d9fb700 1 -- start start 2024-01-21T19:50:53.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.689+0000 7fd18d9fb700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1880f62c0 0x7fd188106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:53.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.689+0000 7fd18d9fb700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd188106a10 con 0x7fd1880f62c0 2024-01-21T19:50:53.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.689+0000 7fd18c9f9700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1880f62c0 0x7fd188106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:53.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.689+0000 7fd18c9f9700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1880f62c0 0x7fd188106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:38768/0 (socket says 172.21.15.173:38768) 2024-01-21T19:50:53.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.689+0000 7fd18c9f9700 1 -- 172.21.15.173:0/409920930 learned_addr learned my addr 172.21.15.173:0/409920930 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:50:53.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.689+0000 7fd18c9f9700 1 -- 172.21.15.173:0/409920930 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c005040 con 0x7fd1880f62c0 2024-01-21T19:50:53.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.689+0000 7fd18c9f9700 1 --2- 172.21.15.173:0/409920930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1880f62c0 0x7fd188106520 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fd17c013510 tx=0x7fd17c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:53.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.689+0000 7fd1857fa700 1 -- 172.21.15.173:0/409920930 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd17c009320 con 0x7fd1880f62c0 2024-01-21T19:50:53.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.690+0000 7fd18d9fb700 1 -- 172.21.15.173:0/409920930 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd188106c10 con 0x7fd1880f62c0 2024-01-21T19:50:53.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.690+0000 7fd1857fa700 1 -- 172.21.15.173:0/409920930 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd17c013f00 con 0x7fd1880f62c0 2024-01-21T19:50:53.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.690+0000 7fd1857fa700 1 -- 172.21.15.173:0/409920930 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd17c014dc0 con 0x7fd1880f62c0 2024-01-21T19:50:53.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.690+0000 7fd18d9fb700 1 -- 172.21.15.173:0/409920930 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1881070a0 con 0x7fd1880f62c0 2024-01-21T19:50:53.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.690+0000 7fd1857fa700 1 -- 172.21.15.173:0/409920930 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd17c0095f0 con 0x7fd1880f62c0 2024-01-21T19:50:53.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.690+0000 7fd18d9fb700 1 -- 172.21.15.173:0/409920930 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd188045ad0 con 0x7fd1880f62c0 2024-01-21T19:50:53.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.690+0000 7fd1857fa700 1 --2- 172.21.15.173:0/409920930 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd17802ff60 0x7fd178032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:53.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.691+0000 7fd1857fa700 1 -- 172.21.15.173:0/409920930 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd17c041080 con 0x7fd1880f62c0 2024-01-21T19:50:53.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.691+0000 7fd187fff700 1 --2- 172.21.15.173:0/409920930 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd17802ff60 0x7fd178032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:53.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.691+0000 7fd187fff700 1 --2- 172.21.15.173:0/409920930 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd17802ff60 0x7fd178032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fd1740096a0 tx=0x7fd174006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:53.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.696+0000 7fd1857fa700 1 -- 172.21.15.173:0/409920930 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd17c01c790 con 0x7fd1880f62c0 2024-01-21T19:50:53.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.901+0000 7fd18d9fb700 1 -- 172.21.15.173:0/409920930 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd188107f50 con 0x7fd1880f62c0 2024-01-21T19:50:53.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.902+0000 7fd1857fa700 1 -- 172.21.15.173:0/409920930 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd17c010230 con 0x7fd1880f62c0 2024-01-21T19:50:53.895 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:50:53.895 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:50:53.896 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.903+0000 7fd18d9fb700 1 -- 172.21.15.173:0/409920930 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd17802ff60 msgr2=0x7fd178032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:53.896 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.903+0000 7fd18d9fb700 1 --2- 172.21.15.173:0/409920930 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd17802ff60 0x7fd178032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fd1740096a0 tx=0x7fd174006b40).stop 2024-01-21T19:50:53.896 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.903+0000 7fd18d9fb700 1 -- 172.21.15.173:0/409920930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1880f62c0 msgr2=0x7fd188106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:53.897 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.903+0000 7fd18d9fb700 1 --2- 172.21.15.173:0/409920930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1880f62c0 0x7fd188106520 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fd17c013510 tx=0x7fd17c00b040).stop 2024-01-21T19:50:53.897 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.903+0000 7fd18d9fb700 1 -- 172.21.15.173:0/409920930 shutdown_connections 2024-01-21T19:50:53.897 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.903+0000 7fd18d9fb700 1 --2- 172.21.15.173:0/409920930 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd17802ff60 0x7fd178032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:53.897 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.903+0000 7fd18d9fb700 1 --2- 172.21.15.173:0/409920930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd1880f62c0 0x7fd188106520 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:53.897 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.903+0000 7fd18d9fb700 1 -- 172.21.15.173:0/409920930 >> 172.21.15.173:0/409920930 conn(0x7fd1880f1ac0 msgr2=0x7fd1880f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:53.897 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.903+0000 7fd18d9fb700 1 -- 172.21.15.173:0/409920930 shutdown_connections 2024-01-21T19:50:53.897 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:53.903+0000 7fd18d9fb700 1 -- 172.21.15.173:0/409920930 wait complete. 2024-01-21T19:50:53.897 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:50:55.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:54 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:52.954558+0000 mgr.smithi169.espgue (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:55.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:54 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:53.903473+0000 mon.smithi169 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.173:0/409920930' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:50:55.282 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:50:55.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:50:56.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.485+0000 7f156d7a6700 1 -- 172.21.15.173:0/3339541942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15680f62a0 msgr2=0x7f15680f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:56.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.485+0000 7f156d7a6700 1 --2- 172.21.15.173:0/3339541942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15680f62a0 0x7f15680f66c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f1558009fb0 tx=0x7f1558009b30).stop 2024-01-21T19:50:56.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.485+0000 7f156d7a6700 1 -- 172.21.15.173:0/3339541942 shutdown_connections 2024-01-21T19:50:56.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.485+0000 7f156d7a6700 1 --2- 172.21.15.173:0/3339541942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15680f62a0 0x7f15680f66c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:56.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.485+0000 7f156d7a6700 1 -- 172.21.15.173:0/3339541942 >> 172.21.15.173:0/3339541942 conn(0x7f15680f1aa0 msgr2=0x7f15680f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:56.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.485+0000 7f156d7a6700 1 -- 172.21.15.173:0/3339541942 shutdown_connections 2024-01-21T19:50:56.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.485+0000 7f156d7a6700 1 -- 172.21.15.173:0/3339541942 wait complete. 2024-01-21T19:50:56.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.485+0000 7f156d7a6700 1 Processor -- start 2024-01-21T19:50:56.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.486+0000 7f156d7a6700 1 -- start start 2024-01-21T19:50:56.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.486+0000 7f156d7a6700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15680f62a0 0x7f156806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:56.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.486+0000 7f156d7a6700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1568069e20 con 0x7f15680f62a0 2024-01-21T19:50:56.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.486+0000 7f1567fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15680f62a0 0x7f156806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:56.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.486+0000 7f1567fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15680f62a0 0x7f156806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:38786/0 (socket says 172.21.15.173:38786) 2024-01-21T19:50:56.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.486+0000 7f1567fff700 1 -- 172.21.15.173:0/3586087438 learned_addr learned my addr 172.21.15.173:0/3586087438 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:50:56.481 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.486+0000 7f1567fff700 1 -- 172.21.15.173:0/3586087438 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1558005040 con 0x7f15680f62a0 2024-01-21T19:50:56.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.487+0000 7f1567fff700 1 --2- 172.21.15.173:0/3586087438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15680f62a0 0x7f156806b9a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f1558013510 tx=0x7f155800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:56.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.487+0000 7f1564ff9700 1 -- 172.21.15.173:0/3586087438 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1558009320 con 0x7f15680f62a0 2024-01-21T19:50:56.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.487+0000 7f156d7a6700 1 -- 172.21.15.173:0/3586087438 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1568069fc0 con 0x7f15680f62a0 2024-01-21T19:50:56.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.487+0000 7f1564ff9700 1 -- 172.21.15.173:0/3586087438 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1558013f00 con 0x7f15680f62a0 2024-01-21T19:50:56.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.487+0000 7f1564ff9700 1 -- 172.21.15.173:0/3586087438 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1558014dc0 con 0x7f15680f62a0 2024-01-21T19:50:56.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.487+0000 7f156d7a6700 1 -- 172.21.15.173:0/3586087438 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f156806a450 con 0x7f15680f62a0 2024-01-21T19:50:56.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.488+0000 7f1564ff9700 1 -- 172.21.15.173:0/3586087438 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f155801d3d0 con 0x7f15680f62a0 2024-01-21T19:50:56.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.488+0000 7f156d7a6700 1 -- 172.21.15.173:0/3586087438 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1568059100 con 0x7f15680f62a0 2024-01-21T19:50:56.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.488+0000 7f1564ff9700 1 --2- 172.21.15.173:0/3586087438 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f155002ff60 0x7f1550032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:56.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.488+0000 7f1564ff9700 1 -- 172.21.15.173:0/3586087438 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1558041050 con 0x7f15680f62a0 2024-01-21T19:50:56.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.488+0000 7f15677fe700 1 --2- 172.21.15.173:0/3586087438 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f155002ff60 0x7f1550032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:56.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.489+0000 7f15677fe700 1 --2- 172.21.15.173:0/3586087438 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f155002ff60 0x7f1550032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f155c0096a0 tx=0x7f155c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:56.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.493+0000 7f1564ff9700 1 -- 172.21.15.173:0/3586087438 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f15580263f0 con 0x7f15680f62a0 2024-01-21T19:50:56.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.699+0000 7f156d7a6700 1 -- 172.21.15.173:0/3586087438 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15680fbce0 con 0x7f15680f62a0 2024-01-21T19:50:56.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.700+0000 7f1564ff9700 1 -- 172.21.15.173:0/3586087438 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1558012330 con 0x7f15680f62a0 2024-01-21T19:50:56.693 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:50:56.693 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:50:56.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.701+0000 7f156d7a6700 1 -- 172.21.15.173:0/3586087438 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f155002ff60 msgr2=0x7f1550032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:56.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.701+0000 7f156d7a6700 1 --2- 172.21.15.173:0/3586087438 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f155002ff60 0x7f1550032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f155c0096a0 tx=0x7f155c006b40).stop 2024-01-21T19:50:56.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.701+0000 7f156d7a6700 1 -- 172.21.15.173:0/3586087438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15680f62a0 msgr2=0x7f156806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:56.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.701+0000 7f156d7a6700 1 --2- 172.21.15.173:0/3586087438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15680f62a0 0x7f156806b9a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f1558013510 tx=0x7f155800b040).stop 2024-01-21T19:50:56.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.702+0000 7f156d7a6700 1 -- 172.21.15.173:0/3586087438 shutdown_connections 2024-01-21T19:50:56.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.702+0000 7f156d7a6700 1 --2- 172.21.15.173:0/3586087438 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f155002ff60 0x7f1550032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:56.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.702+0000 7f156d7a6700 1 --2- 172.21.15.173:0/3586087438 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f15680f62a0 0x7f156806b9a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:56.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.702+0000 7f156d7a6700 1 -- 172.21.15.173:0/3586087438 >> 172.21.15.173:0/3586087438 conn(0x7f15680f1aa0 msgr2=0x7f15680f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:56.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.702+0000 7f156d7a6700 1 -- 172.21.15.173:0/3586087438 shutdown_connections 2024-01-21T19:50:56.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:56.702+0000 7f156d7a6700 1 -- 172.21.15.173:0/3586087438 wait complete. 2024-01-21T19:50:56.696 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:50:57.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:56 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:54.954793+0000 mgr.smithi169.espgue (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:57.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:56 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:56.701495+0000 mon.smithi169 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.173:0/3586087438' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:50:58.088 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:50:58.089 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:50:59.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:56.955045+0000 mgr.smithi169.espgue (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:50:59.271 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.279+0000 7f4de03d7700 1 -- 172.21.15.173:0/2715541135 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dd80f4880 msgr2=0x7f4dd80f4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:59.271 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.279+0000 7f4de03d7700 1 --2- 172.21.15.173:0/2715541135 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dd80f4880 0x7f4dd80f4ca0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f4dd4009fb0 tx=0x7f4dd4009b30).stop 2024-01-21T19:50:59.271 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.279+0000 7f4de03d7700 1 -- 172.21.15.173:0/2715541135 shutdown_connections 2024-01-21T19:50:59.271 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.279+0000 7f4de03d7700 1 --2- 172.21.15.173:0/2715541135 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dd80f4880 0x7f4dd80f4ca0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:59.271 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.279+0000 7f4de03d7700 1 -- 172.21.15.173:0/2715541135 >> 172.21.15.173:0/2715541135 conn(0x7f4dd80f02e0 msgr2=0x7f4dd80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:59.271 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.279+0000 7f4de03d7700 1 -- 172.21.15.173:0/2715541135 shutdown_connections 2024-01-21T19:50:59.271 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.279+0000 7f4de03d7700 1 -- 172.21.15.173:0/2715541135 wait complete. 2024-01-21T19:50:59.273 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.279+0000 7f4de03d7700 1 Processor -- start 2024-01-21T19:50:59.273 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.279+0000 7f4de03d7700 1 -- start start 2024-01-21T19:50:59.273 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.279+0000 7f4de03d7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dd80f4880 0x7f4dd81066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:59.273 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.279+0000 7f4de03d7700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd8106ba0 con 0x7f4dd80f4880 2024-01-21T19:50:59.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.280+0000 7f4dde173700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dd80f4880 0x7f4dd81066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:59.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.280+0000 7f4dde173700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dd80f4880 0x7f4dd81066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:38800/0 (socket says 172.21.15.173:38800) 2024-01-21T19:50:59.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.280+0000 7f4dde173700 1 -- 172.21.15.173:0/1432514855 learned_addr learned my addr 172.21.15.173:0/1432514855 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:50:59.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.280+0000 7f4dde173700 1 -- 172.21.15.173:0/1432514855 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd4005040 con 0x7f4dd80f4880 2024-01-21T19:50:59.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.280+0000 7f4dde173700 1 --2- 172.21.15.173:0/1432514855 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dd80f4880 0x7f4dd81066b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f4dd4013510 tx=0x7f4dd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:59.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.280+0000 7f4dceffd700 1 -- 172.21.15.173:0/1432514855 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4dd4009320 con 0x7f4dd80f4880 2024-01-21T19:50:59.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.280+0000 7f4de03d7700 1 -- 172.21.15.173:0/1432514855 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd8106da0 con 0x7f4dd80f4880 2024-01-21T19:50:59.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.280+0000 7f4dceffd700 1 -- 172.21.15.173:0/1432514855 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4dd4013f00 con 0x7f4dd80f4880 2024-01-21T19:50:59.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.280+0000 7f4dceffd700 1 -- 172.21.15.173:0/1432514855 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4dd4014dc0 con 0x7f4dd80f4880 2024-01-21T19:50:59.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.280+0000 7f4de03d7700 1 -- 172.21.15.173:0/1432514855 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd8107230 con 0x7f4dd80f4880 2024-01-21T19:50:59.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.281+0000 7f4dceffd700 1 -- 172.21.15.173:0/1432514855 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4dd4014420 con 0x7f4dd80f4880 2024-01-21T19:50:59.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.281+0000 7f4de03d7700 1 -- 172.21.15.173:0/1432514855 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd8045ad0 con 0x7f4dd80f4880 2024-01-21T19:50:59.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.281+0000 7f4dceffd700 1 --2- 172.21.15.173:0/1432514855 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4dc402ff60 0x7f4dc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:50:59.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.281+0000 7f4dceffd700 1 -- 172.21.15.173:0/1432514855 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4dd4041110 con 0x7f4dd80f4880 2024-01-21T19:50:59.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.281+0000 7f4ddd972700 1 --2- 172.21.15.173:0/1432514855 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4dc402ff60 0x7f4dc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:50:59.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.282+0000 7f4ddd972700 1 --2- 172.21.15.173:0/1432514855 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4dc402ff60 0x7f4dc4032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f4dc80096a0 tx=0x7f4dc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:50:59.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.286+0000 7f4dceffd700 1 -- 172.21.15.173:0/1432514855 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4dd40095f0 con 0x7f4dd80f4880 2024-01-21T19:50:59.484 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.491+0000 7f4de03d7700 1 -- 172.21.15.173:0/1432514855 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4dd80fbf50 con 0x7f4dd80f4880 2024-01-21T19:50:59.484 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.492+0000 7f4dceffd700 1 -- 172.21.15.173:0/1432514855 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4dd401cb10 con 0x7f4dd80f4880 2024-01-21T19:50:59.485 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:50:59.485 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:50:59.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.494+0000 7f4de03d7700 1 -- 172.21.15.173:0/1432514855 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4dc402ff60 msgr2=0x7f4dc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:59.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.494+0000 7f4de03d7700 1 --2- 172.21.15.173:0/1432514855 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4dc402ff60 0x7f4dc4032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f4dc80096a0 tx=0x7f4dc8006b40).stop 2024-01-21T19:50:59.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.494+0000 7f4de03d7700 1 -- 172.21.15.173:0/1432514855 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dd80f4880 msgr2=0x7f4dd81066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:50:59.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.494+0000 7f4de03d7700 1 --2- 172.21.15.173:0/1432514855 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dd80f4880 0x7f4dd81066b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f4dd4013510 tx=0x7f4dd400b040).stop 2024-01-21T19:50:59.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.494+0000 7f4de03d7700 1 -- 172.21.15.173:0/1432514855 shutdown_connections 2024-01-21T19:50:59.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.494+0000 7f4de03d7700 1 --2- 172.21.15.173:0/1432514855 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4dc402ff60 0x7f4dc4032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:59.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.494+0000 7f4de03d7700 1 --2- 172.21.15.173:0/1432514855 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4dd80f4880 0x7f4dd81066b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:50:59.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.494+0000 7f4de03d7700 1 -- 172.21.15.173:0/1432514855 >> 172.21.15.173:0/1432514855 conn(0x7f4dd80f02e0 msgr2=0x7f4dd80f0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:50:59.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.494+0000 7f4de03d7700 1 -- 172.21.15.173:0/1432514855 shutdown_connections 2024-01-21T19:50:59.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:50:59.494+0000 7f4de03d7700 1 -- 172.21.15.173:0/1432514855 wait complete. 2024-01-21T19:50:59.487 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:00.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:50:59 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:50:59.492842+0000 mon.smithi169 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.173:0/1432514855' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:51:00.892 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:51:00.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:51:01.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:00 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:50:58.955269+0000 mgr.smithi169.espgue (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:02.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.095+0000 7f28248e0700 1 -- 172.21.15.173:0/3877224510 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28200f62a0 msgr2=0x7f28200f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:02.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.095+0000 7f28248e0700 1 --2- 172.21.15.173:0/3877224510 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28200f62a0 0x7f28200f66c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f2810009fb0 tx=0x7f2810009b30).stop 2024-01-21T19:51:02.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.095+0000 7f28248e0700 1 -- 172.21.15.173:0/3877224510 shutdown_connections 2024-01-21T19:51:02.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.095+0000 7f28248e0700 1 --2- 172.21.15.173:0/3877224510 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28200f62a0 0x7f28200f66c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:02.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.095+0000 7f28248e0700 1 -- 172.21.15.173:0/3877224510 >> 172.21.15.173:0/3877224510 conn(0x7f28200f1aa0 msgr2=0x7f28200f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:02.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.095+0000 7f28248e0700 1 -- 172.21.15.173:0/3877224510 shutdown_connections 2024-01-21T19:51:02.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.095+0000 7f28248e0700 1 -- 172.21.15.173:0/3877224510 wait complete. 2024-01-21T19:51:02.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.095+0000 7f28248e0700 1 Processor -- start 2024-01-21T19:51:02.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.095+0000 7f28248e0700 1 -- start start 2024-01-21T19:51:02.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.096+0000 7f28248e0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28200f62a0 0x7f282006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:02.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.096+0000 7f28248e0700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2820069e20 con 0x7f28200f62a0 2024-01-21T19:51:02.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.096+0000 7f281ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28200f62a0 0x7f282006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:02.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.096+0000 7f281ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28200f62a0 0x7f282006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:38824/0 (socket says 172.21.15.173:38824) 2024-01-21T19:51:02.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.096+0000 7f281ed9d700 1 -- 172.21.15.173:0/3365961355 learned_addr learned my addr 172.21.15.173:0/3365961355 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:51:02.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.096+0000 7f281ed9d700 1 -- 172.21.15.173:0/3365961355 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2810005040 con 0x7f28200f62a0 2024-01-21T19:51:02.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.096+0000 7f281ed9d700 1 --2- 172.21.15.173:0/3365961355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28200f62a0 0x7f282006b9a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f2810013510 tx=0x7f281000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:02.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.096+0000 7f28077fe700 1 -- 172.21.15.173:0/3365961355 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2810009320 con 0x7f28200f62a0 2024-01-21T19:51:02.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.096+0000 7f28077fe700 1 -- 172.21.15.173:0/3365961355 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2810013f00 con 0x7f28200f62a0 2024-01-21T19:51:02.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.096+0000 7f28248e0700 1 -- 172.21.15.173:0/3365961355 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2820069fc0 con 0x7f28200f62a0 2024-01-21T19:51:02.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.096+0000 7f28077fe700 1 -- 172.21.15.173:0/3365961355 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2810014dc0 con 0x7f28200f62a0 2024-01-21T19:51:02.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.096+0000 7f28248e0700 1 -- 172.21.15.173:0/3365961355 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f282006a450 con 0x7f28200f62a0 2024-01-21T19:51:02.091 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.097+0000 7f28077fe700 1 -- 172.21.15.173:0/3365961355 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2810014420 con 0x7f28200f62a0 2024-01-21T19:51:02.091 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.097+0000 7f28248e0700 1 -- 172.21.15.173:0/3365961355 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2820059100 con 0x7f28200f62a0 2024-01-21T19:51:02.091 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.097+0000 7f28077fe700 1 --2- 172.21.15.173:0/3365961355 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f280802ff60 0x7f2808032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:02.091 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.097+0000 7f28077fe700 1 -- 172.21.15.173:0/3365961355 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2810041030 con 0x7f28200f62a0 2024-01-21T19:51:02.091 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.097+0000 7f281e59c700 1 --2- 172.21.15.173:0/3365961355 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f280802ff60 0x7f2808032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:02.091 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.098+0000 7f281e59c700 1 --2- 172.21.15.173:0/3365961355 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f280802ff60 0x7f2808032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f28140096a0 tx=0x7f2814006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:02.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.102+0000 7f28077fe700 1 -- 172.21.15.173:0/3365961355 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f28100095f0 con 0x7f28200f62a0 2024-01-21T19:51:02.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.308+0000 7f28248e0700 1 -- 172.21.15.173:0/3365961355 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f28200fbce0 con 0x7f28200f62a0 2024-01-21T19:51:02.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.309+0000 7f28077fe700 1 -- 172.21.15.173:0/3365961355 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f281001cb10 con 0x7f28200f62a0 2024-01-21T19:51:02.301 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:51:02.301 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:51:02.303 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.310+0000 7f28248e0700 1 -- 172.21.15.173:0/3365961355 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f280802ff60 msgr2=0x7f2808032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:02.303 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.310+0000 7f28248e0700 1 --2- 172.21.15.173:0/3365961355 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f280802ff60 0x7f2808032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f28140096a0 tx=0x7f2814006b40).stop 2024-01-21T19:51:02.303 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.310+0000 7f28248e0700 1 -- 172.21.15.173:0/3365961355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28200f62a0 msgr2=0x7f282006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:02.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.310+0000 7f28248e0700 1 --2- 172.21.15.173:0/3365961355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28200f62a0 0x7f282006b9a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f2810013510 tx=0x7f281000b040).stop 2024-01-21T19:51:02.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.310+0000 7f28248e0700 1 -- 172.21.15.173:0/3365961355 shutdown_connections 2024-01-21T19:51:02.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.310+0000 7f28248e0700 1 --2- 172.21.15.173:0/3365961355 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f280802ff60 0x7f2808032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:02.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.311+0000 7f28248e0700 1 --2- 172.21.15.173:0/3365961355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28200f62a0 0x7f282006b9a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:02.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.311+0000 7f28248e0700 1 -- 172.21.15.173:0/3365961355 >> 172.21.15.173:0/3365961355 conn(0x7f28200f1aa0 msgr2=0x7f28200f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:02.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.311+0000 7f28248e0700 1 -- 172.21.15.173:0/3365961355 shutdown_connections 2024-01-21T19:51:02.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:02.311+0000 7f28248e0700 1 -- 172.21.15.173:0/3365961355 wait complete. 2024-01-21T19:51:02.305 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:03.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:02 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:00.955542+0000 mgr.smithi169.espgue (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:03.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:02 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:02.309546+0000 mon.smithi169 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.173:0/3365961355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:51:03.701 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:51:03.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:51:04.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.878+0000 7f2687fff700 1 -- 172.21.15.173:0/1808842686 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26880f51a0 msgr2=0x7f26880f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:04.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.878+0000 7f2687fff700 1 --2- 172.21.15.173:0/1808842686 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26880f51a0 0x7f26880f55c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f2670009fb0 tx=0x7f2670009b30).stop 2024-01-21T19:51:04.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.878+0000 7f2687fff700 1 -- 172.21.15.173:0/1808842686 shutdown_connections 2024-01-21T19:51:04.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.878+0000 7f2687fff700 1 --2- 172.21.15.173:0/1808842686 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26880f51a0 0x7f26880f55c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:04.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.878+0000 7f2687fff700 1 -- 172.21.15.173:0/1808842686 >> 172.21.15.173:0/1808842686 conn(0x7f26880f09a0 msgr2=0x7f26880f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:04.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.878+0000 7f2687fff700 1 -- 172.21.15.173:0/1808842686 shutdown_connections 2024-01-21T19:51:04.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.878+0000 7f2687fff700 1 -- 172.21.15.173:0/1808842686 wait complete. 2024-01-21T19:51:04.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.878+0000 7f2687fff700 1 Processor -- start 2024-01-21T19:51:04.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.878+0000 7f2687fff700 1 -- start start 2024-01-21T19:51:04.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.878+0000 7f2687fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26880f51a0 0x7f26881091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:04.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.878+0000 7f2687fff700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2688060240 con 0x7f26880f51a0 2024-01-21T19:51:04.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.879+0000 7f2686ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26880f51a0 0x7f26881091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:04.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.879+0000 7f2686ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26880f51a0 0x7f26881091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:57940/0 (socket says 172.21.15.173:57940) 2024-01-21T19:51:04.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.879+0000 7f2686ffd700 1 -- 172.21.15.173:0/474764581 learned_addr learned my addr 172.21.15.173:0/474764581 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:51:04.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.879+0000 7f2686ffd700 1 -- 172.21.15.173:0/474764581 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2670005040 con 0x7f26880f51a0 2024-01-21T19:51:04.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.879+0000 7f2686ffd700 1 --2- 172.21.15.173:0/474764581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26880f51a0 0x7f26881091b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f2670013510 tx=0x7f267000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:04.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.879+0000 7f267f7fe700 1 -- 172.21.15.173:0/474764581 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2670009320 con 0x7f26880f51a0 2024-01-21T19:51:04.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.879+0000 7f267f7fe700 1 -- 172.21.15.173:0/474764581 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2670013f00 con 0x7f26880f51a0 2024-01-21T19:51:04.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.879+0000 7f2687fff700 1 -- 172.21.15.173:0/474764581 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2688060440 con 0x7f26880f51a0 2024-01-21T19:51:04.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.879+0000 7f267f7fe700 1 -- 172.21.15.173:0/474764581 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2670014dc0 con 0x7f26880f51a0 2024-01-21T19:51:04.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.879+0000 7f2687fff700 1 -- 172.21.15.173:0/474764581 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2688109860 con 0x7f26880f51a0 2024-01-21T19:51:04.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.880+0000 7f267f7fe700 1 -- 172.21.15.173:0/474764581 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2670014420 con 0x7f26880f51a0 2024-01-21T19:51:04.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.880+0000 7f267f7fe700 1 --2- 172.21.15.173:0/474764581 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f267402ff60 0x7f2674032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:04.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.880+0000 7f267f7fe700 1 -- 172.21.15.173:0/474764581 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2670041110 con 0x7f26880f51a0 2024-01-21T19:51:04.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.880+0000 7f2687fff700 1 -- 172.21.15.173:0/474764581 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2688045ad0 con 0x7f26880f51a0 2024-01-21T19:51:04.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.881+0000 7f26867fc700 1 --2- 172.21.15.173:0/474764581 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f267402ff60 0x7f2674032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:04.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.881+0000 7f26867fc700 1 --2- 172.21.15.173:0/474764581 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f267402ff60 0x7f2674032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f26780096a0 tx=0x7f2678006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:04.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:04.885+0000 7f267f7fe700 1 -- 172.21.15.173:0/474764581 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f26700095f0 con 0x7f26880f51a0 2024-01-21T19:51:05.080 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:05.088+0000 7f2687fff700 1 -- 172.21.15.173:0/474764581 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26880facd0 con 0x7f26880f51a0 2024-01-21T19:51:05.081 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:05.089+0000 7f267f7fe700 1 -- 172.21.15.173:0/474764581 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f267002b020 con 0x7f26880f51a0 2024-01-21T19:51:05.081 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:51:05.081 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:51:05.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:05.090+0000 7f2687fff700 1 -- 172.21.15.173:0/474764581 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f267402ff60 msgr2=0x7f2674032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:05.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:05.090+0000 7f2687fff700 1 --2- 172.21.15.173:0/474764581 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f267402ff60 0x7f2674032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f26780096a0 tx=0x7f2678006b40).stop 2024-01-21T19:51:05.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:05.090+0000 7f2687fff700 1 -- 172.21.15.173:0/474764581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26880f51a0 msgr2=0x7f26881091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:05.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:05.090+0000 7f2687fff700 1 --2- 172.21.15.173:0/474764581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26880f51a0 0x7f26881091b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f2670013510 tx=0x7f267000b040).stop 2024-01-21T19:51:05.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:05.090+0000 7f2687fff700 1 -- 172.21.15.173:0/474764581 shutdown_connections 2024-01-21T19:51:05.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:05.090+0000 7f2687fff700 1 --2- 172.21.15.173:0/474764581 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f267402ff60 0x7f2674032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:05.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:05.090+0000 7f2687fff700 1 --2- 172.21.15.173:0/474764581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f26880f51a0 0x7f26881091b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:05.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:05.090+0000 7f2687fff700 1 -- 172.21.15.173:0/474764581 >> 172.21.15.173:0/474764581 conn(0x7f26880f09a0 msgr2=0x7f26880f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:05.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:05.091+0000 7f2687fff700 1 -- 172.21.15.173:0/474764581 shutdown_connections 2024-01-21T19:51:05.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:05.091+0000 7f2687fff700 1 -- 172.21.15.173:0/474764581 wait complete. 2024-01-21T19:51:05.083 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:05.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:04 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:02.955891+0000 mgr.smithi169.espgue (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:06.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:05 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:05.089557+0000 mon.smithi169 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.173:0/474764581' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:51:06.470 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:51:06.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:51:07.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:06 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:04.956272+0000 mgr.smithi169.espgue (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:07.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.684+0000 7f0eccf55700 1 -- 172.21.15.173:0/3709086355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0ec80f5410 msgr2=0x7f0ec80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:07.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.684+0000 7f0eccf55700 1 --2- 172.21.15.173:0/3709086355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0ec80f5410 0x7f0ec80f5830 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f0eb8009fb0 tx=0x7f0eb8009b30).stop 2024-01-21T19:51:07.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.684+0000 7f0eccf55700 1 -- 172.21.15.173:0/3709086355 shutdown_connections 2024-01-21T19:51:07.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.684+0000 7f0eccf55700 1 --2- 172.21.15.173:0/3709086355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0ec80f5410 0x7f0ec80f5830 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:07.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.684+0000 7f0eccf55700 1 -- 172.21.15.173:0/3709086355 >> 172.21.15.173:0/3709086355 conn(0x7f0ec80f0b70 msgr2=0x7f0ec80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:07.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.684+0000 7f0eccf55700 1 -- 172.21.15.173:0/3709086355 shutdown_connections 2024-01-21T19:51:07.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.684+0000 7f0eccf55700 1 -- 172.21.15.173:0/3709086355 wait complete. 2024-01-21T19:51:07.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.685+0000 7f0eccf55700 1 Processor -- start 2024-01-21T19:51:07.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.685+0000 7f0eccf55700 1 -- start start 2024-01-21T19:51:07.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.685+0000 7f0eccf55700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0ec80f5410 0x7f0ec8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:07.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.685+0000 7f0eccf55700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec8105b20 con 0x7f0ec80f5410 2024-01-21T19:51:07.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.686+0000 7f0ec659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0ec80f5410 0x7f0ec8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:07.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.686+0000 7f0ec659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0ec80f5410 0x7f0ec8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:57964/0 (socket says 172.21.15.173:57964) 2024-01-21T19:51:07.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.686+0000 7f0ec659c700 1 -- 172.21.15.173:0/4262532904 learned_addr learned my addr 172.21.15.173:0/4262532904 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:51:07.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.686+0000 7f0ec659c700 1 -- 172.21.15.173:0/4262532904 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb8005040 con 0x7f0ec80f5410 2024-01-21T19:51:07.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.686+0000 7f0ec659c700 1 --2- 172.21.15.173:0/4262532904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0ec80f5410 0x7f0ec8105630 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f0eb8013510 tx=0x7f0eb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:07.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.686+0000 7f0eb6ffd700 1 -- 172.21.15.173:0/4262532904 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0eb8009320 con 0x7f0ec80f5410 2024-01-21T19:51:07.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.686+0000 7f0eb6ffd700 1 -- 172.21.15.173:0/4262532904 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0eb8013f00 con 0x7f0ec80f5410 2024-01-21T19:51:07.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.686+0000 7f0eccf55700 1 -- 172.21.15.173:0/4262532904 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec8105d20 con 0x7f0ec80f5410 2024-01-21T19:51:07.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.686+0000 7f0eb6ffd700 1 -- 172.21.15.173:0/4262532904 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0eb8014d90 con 0x7f0ec80f5410 2024-01-21T19:51:07.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.686+0000 7f0eccf55700 1 -- 172.21.15.173:0/4262532904 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec81061b0 con 0x7f0ec80f5410 2024-01-21T19:51:07.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.687+0000 7f0eb6ffd700 1 -- 172.21.15.173:0/4262532904 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0eb8014420 con 0x7f0ec80f5410 2024-01-21T19:51:07.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.687+0000 7f0eccf55700 1 -- 172.21.15.173:0/4262532904 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec8045ad0 con 0x7f0ec80f5410 2024-01-21T19:51:07.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.687+0000 7f0eb6ffd700 1 --2- 172.21.15.173:0/4262532904 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0eb002ff60 0x7f0eb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:07.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.687+0000 7f0eb6ffd700 1 -- 172.21.15.173:0/4262532904 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0eb80410c0 con 0x7f0ec80f5410 2024-01-21T19:51:07.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.687+0000 7f0ec5d9b700 1 --2- 172.21.15.173:0/4262532904 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0eb002ff60 0x7f0eb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:07.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.688+0000 7f0ec5d9b700 1 --2- 172.21.15.173:0/4262532904 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0eb002ff60 0x7f0eb0032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f0ebc0096a0 tx=0x7f0ebc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:07.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.692+0000 7f0eb6ffd700 1 -- 172.21.15.173:0/4262532904 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0eb80095f0 con 0x7f0ec80f5410 2024-01-21T19:51:07.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.897+0000 7f0eccf55700 1 -- 172.21.15.173:0/4262532904 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ec80fae60 con 0x7f0ec80f5410 2024-01-21T19:51:07.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.898+0000 7f0eb6ffd700 1 -- 172.21.15.173:0/4262532904 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0eb801db70 con 0x7f0ec80f5410 2024-01-21T19:51:07.890 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:51:07.890 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:51:07.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.899+0000 7f0eccf55700 1 -- 172.21.15.173:0/4262532904 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0eb002ff60 msgr2=0x7f0eb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:07.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.899+0000 7f0eccf55700 1 --2- 172.21.15.173:0/4262532904 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0eb002ff60 0x7f0eb0032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f0ebc0096a0 tx=0x7f0ebc006b40).stop 2024-01-21T19:51:07.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.899+0000 7f0eccf55700 1 -- 172.21.15.173:0/4262532904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0ec80f5410 msgr2=0x7f0ec8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:07.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.899+0000 7f0eccf55700 1 --2- 172.21.15.173:0/4262532904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0ec80f5410 0x7f0ec8105630 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f0eb8013510 tx=0x7f0eb800b040).stop 2024-01-21T19:51:07.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.899+0000 7f0eccf55700 1 -- 172.21.15.173:0/4262532904 shutdown_connections 2024-01-21T19:51:07.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.899+0000 7f0eccf55700 1 --2- 172.21.15.173:0/4262532904 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0eb002ff60 0x7f0eb0032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:07.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.899+0000 7f0eccf55700 1 --2- 172.21.15.173:0/4262532904 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0ec80f5410 0x7f0ec8105630 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:07.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.899+0000 7f0eccf55700 1 -- 172.21.15.173:0/4262532904 >> 172.21.15.173:0/4262532904 conn(0x7f0ec80f0b70 msgr2=0x7f0ec80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:07.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.899+0000 7f0eccf55700 1 -- 172.21.15.173:0/4262532904 shutdown_connections 2024-01-21T19:51:07.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:07.899+0000 7f0eccf55700 1 -- 172.21.15.173:0/4262532904 wait complete. 2024-01-21T19:51:07.893 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:09.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:08 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:06.956619+0000 mgr.smithi169.espgue (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:09.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:08 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:07.898557+0000 mon.smithi169 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.173:0/4262532904' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:51:09.277 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:51:09.278 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:51:10.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.482+0000 7fac8fd59700 1 -- 172.21.15.173:0/2427297581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fac880f6510 msgr2=0x7fac880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:10.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.482+0000 7fac8fd59700 1 --2- 172.21.15.173:0/2427297581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fac880f6510 0x7fac880f6930 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fac78009fb0 tx=0x7fac78009b30).stop 2024-01-21T19:51:10.478 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.482+0000 7fac8fd59700 1 -- 172.21.15.173:0/2427297581 shutdown_connections 2024-01-21T19:51:10.478 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.482+0000 7fac8fd59700 1 --2- 172.21.15.173:0/2427297581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fac880f6510 0x7fac880f6930 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:10.478 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.482+0000 7fac8fd59700 1 -- 172.21.15.173:0/2427297581 >> 172.21.15.173:0/2427297581 conn(0x7fac880f1c70 msgr2=0x7fac880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:10.478 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.482+0000 7fac8fd59700 1 -- 172.21.15.173:0/2427297581 shutdown_connections 2024-01-21T19:51:10.478 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.482+0000 7fac8fd59700 1 -- 172.21.15.173:0/2427297581 wait complete. 2024-01-21T19:51:10.478 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.483+0000 7fac8fd59700 1 Processor -- start 2024-01-21T19:51:10.478 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.483+0000 7fac8fd59700 1 -- start start 2024-01-21T19:51:10.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.483+0000 7fac8fd59700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fac880f6510 0x7fac88106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:10.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.483+0000 7fac8fd59700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac88106c20 con 0x7fac880f6510 2024-01-21T19:51:10.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.483+0000 7fac8daf5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fac880f6510 0x7fac88106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:10.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.483+0000 7fac8daf5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fac880f6510 0x7fac88106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:57972/0 (socket says 172.21.15.173:57972) 2024-01-21T19:51:10.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.483+0000 7fac8daf5700 1 -- 172.21.15.173:0/2670436974 learned_addr learned my addr 172.21.15.173:0/2670436974 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:51:10.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.483+0000 7fac8daf5700 1 -- 172.21.15.173:0/2670436974 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac78005040 con 0x7fac880f6510 2024-01-21T19:51:10.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.484+0000 7fac8daf5700 1 --2- 172.21.15.173:0/2670436974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fac880f6510 0x7fac88106730 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fac78013510 tx=0x7fac7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:10.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.484+0000 7fac7e7fc700 1 -- 172.21.15.173:0/2670436974 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac78009320 con 0x7fac880f6510 2024-01-21T19:51:10.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.484+0000 7fac8fd59700 1 -- 172.21.15.173:0/2670436974 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac88106e20 con 0x7fac880f6510 2024-01-21T19:51:10.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.484+0000 7fac7e7fc700 1 -- 172.21.15.173:0/2670436974 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac78013f00 con 0x7fac880f6510 2024-01-21T19:51:10.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.484+0000 7fac7e7fc700 1 -- 172.21.15.173:0/2670436974 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac78014d90 con 0x7fac880f6510 2024-01-21T19:51:10.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.484+0000 7fac8fd59700 1 -- 172.21.15.173:0/2670436974 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac881072b0 con 0x7fac880f6510 2024-01-21T19:51:10.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.485+0000 7fac7e7fc700 1 -- 172.21.15.173:0/2670436974 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fac780095f0 con 0x7fac880f6510 2024-01-21T19:51:10.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.485+0000 7fac8fd59700 1 -- 172.21.15.173:0/2670436974 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac88045ad0 con 0x7fac880f6510 2024-01-21T19:51:10.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.485+0000 7fac7e7fc700 1 --2- 172.21.15.173:0/2670436974 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fac7402ff60 0x7fac74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:10.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.485+0000 7fac7e7fc700 1 -- 172.21.15.173:0/2670436974 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fac78040fc0 con 0x7fac880f6510 2024-01-21T19:51:10.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.485+0000 7fac8d2f4700 1 --2- 172.21.15.173:0/2670436974 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fac7402ff60 0x7fac74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:10.481 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.485+0000 7fac8d2f4700 1 --2- 172.21.15.173:0/2670436974 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fac7402ff60 0x7fac74032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fac840096a0 tx=0x7fac84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:10.481 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.489+0000 7fac7e7fc700 1 -- 172.21.15.173:0/2670436974 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac78014420 con 0x7fac880f6510 2024-01-21T19:51:10.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.692+0000 7fac8fd59700 1 -- 172.21.15.173:0/2670436974 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac880fbf60 con 0x7fac880f6510 2024-01-21T19:51:10.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.693+0000 7fac7e7fc700 1 -- 172.21.15.173:0/2670436974 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fac78010230 con 0x7fac880f6510 2024-01-21T19:51:10.685 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:51:10.685 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:51:10.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.694+0000 7fac8fd59700 1 -- 172.21.15.173:0/2670436974 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fac7402ff60 msgr2=0x7fac74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:10.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.694+0000 7fac8fd59700 1 --2- 172.21.15.173:0/2670436974 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fac7402ff60 0x7fac74032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fac840096a0 tx=0x7fac84006b40).stop 2024-01-21T19:51:10.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.694+0000 7fac8fd59700 1 -- 172.21.15.173:0/2670436974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fac880f6510 msgr2=0x7fac88106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:10.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.694+0000 7fac8fd59700 1 --2- 172.21.15.173:0/2670436974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fac880f6510 0x7fac88106730 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fac78013510 tx=0x7fac7800b040).stop 2024-01-21T19:51:10.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.694+0000 7fac8fd59700 1 -- 172.21.15.173:0/2670436974 shutdown_connections 2024-01-21T19:51:10.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.694+0000 7fac8fd59700 1 --2- 172.21.15.173:0/2670436974 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fac7402ff60 0x7fac74032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:10.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.694+0000 7fac8fd59700 1 --2- 172.21.15.173:0/2670436974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fac880f6510 0x7fac88106730 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:10.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.695+0000 7fac8fd59700 1 -- 172.21.15.173:0/2670436974 >> 172.21.15.173:0/2670436974 conn(0x7fac880f1c70 msgr2=0x7fac880f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:10.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.695+0000 7fac8fd59700 1 -- 172.21.15.173:0/2670436974 shutdown_connections 2024-01-21T19:51:10.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:10.695+0000 7fac8fd59700 1 -- 172.21.15.173:0/2670436974 wait complete. 2024-01-21T19:51:10.688 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:11.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:10 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:08.956928+0000 mgr.smithi169.espgue (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:11.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:10 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:10.693593+0000 mon.smithi169 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.173:0/2670436974' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:51:12.074 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:51:12.075 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:51:13.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:12 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:10.957254+0000 mgr.smithi169.espgue (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:13.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.313+0000 7f20daedd700 1 -- 172.21.15.173:0/3648649311 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20d40f51a0 msgr2=0x7f20d40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:13.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.313+0000 7f20daedd700 1 --2- 172.21.15.173:0/3648649311 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20d40f51a0 0x7f20d40f55c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f20c4009fb0 tx=0x7f20c4009b30).stop 2024-01-21T19:51:13.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.313+0000 7f20daedd700 1 -- 172.21.15.173:0/3648649311 shutdown_connections 2024-01-21T19:51:13.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.313+0000 7f20daedd700 1 --2- 172.21.15.173:0/3648649311 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20d40f51a0 0x7f20d40f55c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:13.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.313+0000 7f20daedd700 1 -- 172.21.15.173:0/3648649311 >> 172.21.15.173:0/3648649311 conn(0x7f20d40f09a0 msgr2=0x7f20d40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:13.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.313+0000 7f20daedd700 1 -- 172.21.15.173:0/3648649311 shutdown_connections 2024-01-21T19:51:13.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.313+0000 7f20daedd700 1 -- 172.21.15.173:0/3648649311 wait complete. 2024-01-21T19:51:13.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.313+0000 7f20daedd700 1 Processor -- start 2024-01-21T19:51:13.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.313+0000 7f20daedd700 1 -- start start 2024-01-21T19:51:13.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.313+0000 7f20daedd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20d40f51a0 0x7f20d41064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:13.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.313+0000 7f20daedd700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d4106990 con 0x7f20d40f51a0 2024-01-21T19:51:13.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.314+0000 7f20d9edb700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20d40f51a0 0x7f20d41064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:13.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.314+0000 7f20d9edb700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20d40f51a0 0x7f20d41064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:46494/0 (socket says 172.21.15.173:46494) 2024-01-21T19:51:13.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.314+0000 7f20d9edb700 1 -- 172.21.15.173:0/4087053195 learned_addr learned my addr 172.21.15.173:0/4087053195 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:51:13.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.314+0000 7f20d9edb700 1 -- 172.21.15.173:0/4087053195 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c4005040 con 0x7f20d40f51a0 2024-01-21T19:51:13.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.314+0000 7f20d9edb700 1 --2- 172.21.15.173:0/4087053195 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20d40f51a0 0x7f20d41064a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f20c4013510 tx=0x7f20c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:13.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.314+0000 7f20ca7fc700 1 -- 172.21.15.173:0/4087053195 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20c4009320 con 0x7f20d40f51a0 2024-01-21T19:51:13.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.314+0000 7f20daedd700 1 -- 172.21.15.173:0/4087053195 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d4106b90 con 0x7f20d40f51a0 2024-01-21T19:51:13.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.314+0000 7f20ca7fc700 1 -- 172.21.15.173:0/4087053195 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20c4013f00 con 0x7f20d40f51a0 2024-01-21T19:51:13.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.314+0000 7f20ca7fc700 1 -- 172.21.15.173:0/4087053195 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20c4014dc0 con 0x7f20d40f51a0 2024-01-21T19:51:13.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.314+0000 7f20daedd700 1 -- 172.21.15.173:0/4087053195 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d4107020 con 0x7f20d40f51a0 2024-01-21T19:51:13.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.315+0000 7f20ca7fc700 1 -- 172.21.15.173:0/4087053195 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f20c4014420 con 0x7f20d40f51a0 2024-01-21T19:51:13.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.315+0000 7f20daedd700 1 -- 172.21.15.173:0/4087053195 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d4045ad0 con 0x7f20d40f51a0 2024-01-21T19:51:13.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.315+0000 7f20ca7fc700 1 --2- 172.21.15.173:0/4087053195 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f20c002ff60 0x7f20c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:13.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.315+0000 7f20ca7fc700 1 -- 172.21.15.173:0/4087053195 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f20c4041050 con 0x7f20d40f51a0 2024-01-21T19:51:13.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.316+0000 7f20d96da700 1 --2- 172.21.15.173:0/4087053195 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f20c002ff60 0x7f20c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:13.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.316+0000 7f20d96da700 1 --2- 172.21.15.173:0/4087053195 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f20c002ff60 0x7f20c0032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f20d00096a0 tx=0x7f20d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:13.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.320+0000 7f20ca7fc700 1 -- 172.21.15.173:0/4087053195 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f20c40095f0 con 0x7f20d40f51a0 2024-01-21T19:51:13.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.524+0000 7f20daedd700 1 -- 172.21.15.173:0/4087053195 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20d4107ed0 con 0x7f20d40f51a0 2024-01-21T19:51:13.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.525+0000 7f20ca7fc700 1 -- 172.21.15.173:0/4087053195 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f20c401cb10 con 0x7f20d40f51a0 2024-01-21T19:51:13.518 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:51:13.518 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:51:13.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.526+0000 7f20daedd700 1 -- 172.21.15.173:0/4087053195 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f20c002ff60 msgr2=0x7f20c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:13.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.526+0000 7f20daedd700 1 --2- 172.21.15.173:0/4087053195 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f20c002ff60 0x7f20c0032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f20d00096a0 tx=0x7f20d0006b40).stop 2024-01-21T19:51:13.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.526+0000 7f20daedd700 1 -- 172.21.15.173:0/4087053195 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20d40f51a0 msgr2=0x7f20d41064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:13.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.526+0000 7f20daedd700 1 --2- 172.21.15.173:0/4087053195 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20d40f51a0 0x7f20d41064a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f20c4013510 tx=0x7f20c400b040).stop 2024-01-21T19:51:13.520 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.526+0000 7f20daedd700 1 -- 172.21.15.173:0/4087053195 shutdown_connections 2024-01-21T19:51:13.520 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.526+0000 7f20daedd700 1 --2- 172.21.15.173:0/4087053195 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f20c002ff60 0x7f20c0032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:13.520 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.526+0000 7f20daedd700 1 --2- 172.21.15.173:0/4087053195 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20d40f51a0 0x7f20d41064a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:13.520 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.526+0000 7f20daedd700 1 -- 172.21.15.173:0/4087053195 >> 172.21.15.173:0/4087053195 conn(0x7f20d40f09a0 msgr2=0x7f20d405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:13.520 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.526+0000 7f20daedd700 1 -- 172.21.15.173:0/4087053195 shutdown_connections 2024-01-21T19:51:13.520 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:13.526+0000 7f20daedd700 1 -- 172.21.15.173:0/4087053195 wait complete. 2024-01-21T19:51:13.520 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:14.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:13 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:13.525440+0000 mon.smithi169 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.173:0/4087053195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:51:14.907 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:51:14.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:51:15.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:14 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:12.957611+0000 mgr.smithi169.espgue (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:16.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.145+0000 7f33e223f700 1 -- 172.21.15.173:0/3039682349 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33dc0f6510 msgr2=0x7f33dc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:16.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.145+0000 7f33e223f700 1 --2- 172.21.15.173:0/3039682349 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33dc0f6510 0x7f33dc0f6930 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f33c4009fb0 tx=0x7f33c4009b30).stop 2024-01-21T19:51:16.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.145+0000 7f33e223f700 1 -- 172.21.15.173:0/3039682349 shutdown_connections 2024-01-21T19:51:16.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.145+0000 7f33e223f700 1 --2- 172.21.15.173:0/3039682349 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33dc0f6510 0x7f33dc0f6930 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:16.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.145+0000 7f33e223f700 1 -- 172.21.15.173:0/3039682349 >> 172.21.15.173:0/3039682349 conn(0x7f33dc0f1c70 msgr2=0x7f33dc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:16.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.145+0000 7f33e223f700 1 -- 172.21.15.173:0/3039682349 shutdown_connections 2024-01-21T19:51:16.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.145+0000 7f33e223f700 1 -- 172.21.15.173:0/3039682349 wait complete. 2024-01-21T19:51:16.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.146+0000 7f33e223f700 1 Processor -- start 2024-01-21T19:51:16.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.146+0000 7f33e223f700 1 -- start start 2024-01-21T19:51:16.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.146+0000 7f33e223f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33dc108a60 0x7f33dc108e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:16.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.146+0000 7f33e223f700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33dc109370 con 0x7f33dc108a60 2024-01-21T19:51:16.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.147+0000 7f33db7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33dc108a60 0x7f33dc108e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:16.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.147+0000 7f33db7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33dc108a60 0x7f33dc108e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:46508/0 (socket says 172.21.15.173:46508) 2024-01-21T19:51:16.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.147+0000 7f33db7fe700 1 -- 172.21.15.173:0/805396337 learned_addr learned my addr 172.21.15.173:0/805396337 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:51:16.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.147+0000 7f33db7fe700 1 -- 172.21.15.173:0/805396337 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c4005040 con 0x7f33dc108a60 2024-01-21T19:51:16.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.147+0000 7f33db7fe700 1 --2- 172.21.15.173:0/805396337 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33dc108a60 0x7f33dc108e80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f33c4009f80 tx=0x7f33c4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:16.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.147+0000 7f33d3fff700 1 -- 172.21.15.173:0/805396337 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c40136c0 con 0x7f33dc108a60 2024-01-21T19:51:16.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.147+0000 7f33e223f700 1 -- 172.21.15.173:0/805396337 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33dc109570 con 0x7f33dc108a60 2024-01-21T19:51:16.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.147+0000 7f33d3fff700 1 -- 172.21.15.173:0/805396337 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33c40095f0 con 0x7f33dc108a60 2024-01-21T19:51:16.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.147+0000 7f33d3fff700 1 -- 172.21.15.173:0/805396337 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c4014420 con 0x7f33dc108a60 2024-01-21T19:51:16.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.147+0000 7f33e223f700 1 -- 172.21.15.173:0/805396337 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33dc10a0c0 con 0x7f33dc108a60 2024-01-21T19:51:16.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.148+0000 7f33d3fff700 1 -- 172.21.15.173:0/805396337 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f33c4025060 con 0x7f33dc108a60 2024-01-21T19:51:16.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.148+0000 7f33e223f700 1 -- 172.21.15.173:0/805396337 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33dc059100 con 0x7f33dc108a60 2024-01-21T19:51:16.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.148+0000 7f33d3fff700 1 --2- 172.21.15.173:0/805396337 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f33c802ffb0 0x7f33c8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:16.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.148+0000 7f33d3fff700 1 -- 172.21.15.173:0/805396337 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f33c4040e70 con 0x7f33dc108a60 2024-01-21T19:51:16.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.149+0000 7f33daffd700 1 --2- 172.21.15.173:0/805396337 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f33c802ffb0 0x7f33c8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:16.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.149+0000 7f33daffd700 1 --2- 172.21.15.173:0/805396337 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f33c802ffb0 0x7f33c8032470 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f33cc0096a0 tx=0x7f33cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:16.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.153+0000 7f33d3fff700 1 -- 172.21.15.173:0/805396337 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f33c400f280 con 0x7f33dc108a60 2024-01-21T19:51:16.348 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.355+0000 7f33e223f700 1 -- 172.21.15.173:0/805396337 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f33dc045ad0 con 0x7f33dc108a60 2024-01-21T19:51:16.348 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.356+0000 7f33d3fff700 1 -- 172.21.15.173:0/805396337 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f33c402d2d0 con 0x7f33dc108a60 2024-01-21T19:51:16.348 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:51:16.349 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:51:16.350 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.358+0000 7f33e223f700 1 -- 172.21.15.173:0/805396337 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f33c802ffb0 msgr2=0x7f33c8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:16.350 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.358+0000 7f33e223f700 1 --2- 172.21.15.173:0/805396337 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f33c802ffb0 0x7f33c8032470 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f33cc0096a0 tx=0x7f33cc006b40).stop 2024-01-21T19:51:16.350 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.358+0000 7f33e223f700 1 -- 172.21.15.173:0/805396337 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33dc108a60 msgr2=0x7f33dc108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:16.350 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.358+0000 7f33e223f700 1 --2- 172.21.15.173:0/805396337 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33dc108a60 0x7f33dc108e80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f33c4009f80 tx=0x7f33c4009b30).stop 2024-01-21T19:51:16.350 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.358+0000 7f33e223f700 1 -- 172.21.15.173:0/805396337 shutdown_connections 2024-01-21T19:51:16.350 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.358+0000 7f33e223f700 1 --2- 172.21.15.173:0/805396337 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f33c802ffb0 0x7f33c8032470 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:16.350 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.358+0000 7f33e223f700 1 --2- 172.21.15.173:0/805396337 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33dc108a60 0x7f33dc108e80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:16.351 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.358+0000 7f33e223f700 1 -- 172.21.15.173:0/805396337 >> 172.21.15.173:0/805396337 conn(0x7f33dc0f1c70 msgr2=0x7f33dc0f2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:16.351 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.358+0000 7f33e223f700 1 -- 172.21.15.173:0/805396337 shutdown_connections 2024-01-21T19:51:16.351 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:16.358+0000 7f33e223f700 1 -- 172.21.15.173:0/805396337 wait complete. 2024-01-21T19:51:16.351 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:17.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:16 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:14.957890+0000 mgr.smithi169.espgue (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:17.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:16 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:16.356783+0000 mon.smithi169 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.173:0/805396337' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:51:17.743 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:51:17.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:51:18.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.976+0000 7f1c86e38700 1 -- 172.21.15.173:0/374577504 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c800f47c0 msgr2=0x7f1c800f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:18.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.976+0000 7f1c86e38700 1 --2- 172.21.15.173:0/374577504 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c800f47c0 0x7f1c800f4be0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f1c74009fb0 tx=0x7f1c74009b30).stop 2024-01-21T19:51:18.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.976+0000 7f1c86e38700 1 -- 172.21.15.173:0/374577504 shutdown_connections 2024-01-21T19:51:18.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.976+0000 7f1c86e38700 1 --2- 172.21.15.173:0/374577504 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c800f47c0 0x7f1c800f4be0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:18.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.976+0000 7f1c86e38700 1 -- 172.21.15.173:0/374577504 >> 172.21.15.173:0/374577504 conn(0x7f1c800f02e0 msgr2=0x7f1c800f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:18.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.976+0000 7f1c86e38700 1 -- 172.21.15.173:0/374577504 shutdown_connections 2024-01-21T19:51:18.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.976+0000 7f1c86e38700 1 -- 172.21.15.173:0/374577504 wait complete. 2024-01-21T19:51:18.970 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.977+0000 7f1c86e38700 1 Processor -- start 2024-01-21T19:51:18.970 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.977+0000 7f1c86e38700 1 -- start start 2024-01-21T19:51:18.971 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.977+0000 7f1c86e38700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c800f47c0 0x7f1c80106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:18.971 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.977+0000 7f1c86e38700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c80106bf0 con 0x7f1c800f47c0 2024-01-21T19:51:18.971 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.978+0000 7f1c84bd4700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c800f47c0 0x7f1c80106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:18.971 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.978+0000 7f1c84bd4700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c800f47c0 0x7f1c80106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:46522/0 (socket says 172.21.15.173:46522) 2024-01-21T19:51:18.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.978+0000 7f1c84bd4700 1 -- 172.21.15.173:0/3809769490 learned_addr learned my addr 172.21.15.173:0/3809769490 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:51:18.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.978+0000 7f1c84bd4700 1 -- 172.21.15.173:0/3809769490 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c74005040 con 0x7f1c800f47c0 2024-01-21T19:51:18.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.978+0000 7f1c84bd4700 1 --2- 172.21.15.173:0/3809769490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c800f47c0 0x7f1c80106700 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f1c74013510 tx=0x7f1c7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:18.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.978+0000 7f1c7d7fa700 1 -- 172.21.15.173:0/3809769490 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c74009320 con 0x7f1c800f47c0 2024-01-21T19:51:18.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.978+0000 7f1c7d7fa700 1 -- 172.21.15.173:0/3809769490 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c74013f00 con 0x7f1c800f47c0 2024-01-21T19:51:18.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.978+0000 7f1c86e38700 1 -- 172.21.15.173:0/3809769490 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c80106df0 con 0x7f1c800f47c0 2024-01-21T19:51:18.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.978+0000 7f1c7d7fa700 1 -- 172.21.15.173:0/3809769490 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c74014d90 con 0x7f1c800f47c0 2024-01-21T19:51:18.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.978+0000 7f1c86e38700 1 -- 172.21.15.173:0/3809769490 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c80107280 con 0x7f1c800f47c0 2024-01-21T19:51:18.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.979+0000 7f1c7d7fa700 1 -- 172.21.15.173:0/3809769490 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1c740263a0 con 0x7f1c800f47c0 2024-01-21T19:51:18.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.979+0000 7f1c7d7fa700 1 --2- 172.21.15.173:0/3809769490 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1c7002ff60 0x7f1c70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:18.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.979+0000 7f1c86e38700 1 -- 172.21.15.173:0/3809769490 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c80045ad0 con 0x7f1c800f47c0 2024-01-21T19:51:18.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.979+0000 7f1c7d7fa700 1 -- 172.21.15.173:0/3809769490 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1c740413a0 con 0x7f1c800f47c0 2024-01-21T19:51:18.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.979+0000 7f1c7ffff700 1 --2- 172.21.15.173:0/3809769490 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1c7002ff60 0x7f1c70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:18.973 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.980+0000 7f1c7ffff700 1 --2- 172.21.15.173:0/3809769490 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1c7002ff60 0x7f1c70032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f1c6c0096a0 tx=0x7f1c6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:18.976 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:18.984+0000 7f1c7d7fa700 1 -- 172.21.15.173:0/3809769490 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1c7401ccf0 con 0x7f1c800f47c0 2024-01-21T19:51:19.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:18 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:16.958242+0000 mgr.smithi169.espgue (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:19.177 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:19.185+0000 7f1c86e38700 1 -- 172.21.15.173:0/3809769490 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c80108130 con 0x7f1c800f47c0 2024-01-21T19:51:19.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:19.186+0000 7f1c7d7fa700 1 -- 172.21.15.173:0/3809769490 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1c7401cb10 con 0x7f1c800f47c0 2024-01-21T19:51:19.178 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:51:19.178 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:51:19.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:19.187+0000 7f1c86e38700 1 -- 172.21.15.173:0/3809769490 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1c7002ff60 msgr2=0x7f1c70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:19.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:19.187+0000 7f1c86e38700 1 --2- 172.21.15.173:0/3809769490 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1c7002ff60 0x7f1c70032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f1c6c0096a0 tx=0x7f1c6c006b40).stop 2024-01-21T19:51:19.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:19.187+0000 7f1c86e38700 1 -- 172.21.15.173:0/3809769490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c800f47c0 msgr2=0x7f1c80106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:19.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:19.187+0000 7f1c86e38700 1 --2- 172.21.15.173:0/3809769490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c800f47c0 0x7f1c80106700 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f1c74013510 tx=0x7f1c7400b040).stop 2024-01-21T19:51:19.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:19.187+0000 7f1c86e38700 1 -- 172.21.15.173:0/3809769490 shutdown_connections 2024-01-21T19:51:19.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:19.187+0000 7f1c86e38700 1 --2- 172.21.15.173:0/3809769490 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1c7002ff60 0x7f1c70032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:19.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:19.187+0000 7f1c86e38700 1 --2- 172.21.15.173:0/3809769490 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1c800f47c0 0x7f1c80106700 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:19.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:19.187+0000 7f1c86e38700 1 -- 172.21.15.173:0/3809769490 >> 172.21.15.173:0/3809769490 conn(0x7f1c800f02e0 msgr2=0x7f1c800f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:19.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:19.187+0000 7f1c86e38700 1 -- 172.21.15.173:0/3809769490 shutdown_connections 2024-01-21T19:51:19.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:19.187+0000 7f1c86e38700 1 -- 172.21.15.173:0/3809769490 wait complete. 2024-01-21T19:51:19.183 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:20.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:19 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:19.186442+0000 mon.smithi169 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.173:0/3809769490' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:51:20.577 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:51:20.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:51:21.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:20 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:18.958554+0000 mgr.smithi169.espgue (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:21.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.807+0000 7fd9292f5700 1 -- 172.21.15.173:0/1131770926 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd924060280 msgr2=0x7fd9240f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:21.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.807+0000 7fd9292f5700 1 --2- 172.21.15.173:0/1131770926 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd924060280 0x7fd9240f90f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fd914009fb0 tx=0x7fd914009b30).stop 2024-01-21T19:51:21.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.807+0000 7fd9292f5700 1 -- 172.21.15.173:0/1131770926 shutdown_connections 2024-01-21T19:51:21.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.807+0000 7fd9292f5700 1 --2- 172.21.15.173:0/1131770926 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd924060280 0x7fd9240f90f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:21.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.807+0000 7fd9292f5700 1 -- 172.21.15.173:0/1131770926 >> 172.21.15.173:0/1131770926 conn(0x7fd9240f0b70 msgr2=0x7fd9240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:21.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.808+0000 7fd9292f5700 1 -- 172.21.15.173:0/1131770926 shutdown_connections 2024-01-21T19:51:21.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.808+0000 7fd9292f5700 1 -- 172.21.15.173:0/1131770926 wait complete. 2024-01-21T19:51:21.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.808+0000 7fd9292f5700 1 Processor -- start 2024-01-21T19:51:21.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.808+0000 7fd9292f5700 1 -- start start 2024-01-21T19:51:21.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.808+0000 7fd9292f5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd924060280 0x7fd9241066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:21.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.808+0000 7fd9292f5700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd924106bb0 con 0x7fd924060280 2024-01-21T19:51:21.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.809+0000 7fd922ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd924060280 0x7fd9241066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:21.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.809+0000 7fd922ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd924060280 0x7fd9241066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:46532/0 (socket says 172.21.15.173:46532) 2024-01-21T19:51:21.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.809+0000 7fd922ffd700 1 -- 172.21.15.173:0/1840653932 learned_addr learned my addr 172.21.15.173:0/1840653932 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:51:21.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.809+0000 7fd922ffd700 1 -- 172.21.15.173:0/1840653932 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd914005040 con 0x7fd924060280 2024-01-21T19:51:21.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.809+0000 7fd922ffd700 1 --2- 172.21.15.173:0/1840653932 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd924060280 0x7fd9241066c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fd914009830 tx=0x7fd91400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:21.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.809+0000 7fd91b7fe700 1 -- 172.21.15.173:0/1840653932 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd914009320 con 0x7fd924060280 2024-01-21T19:51:21.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.809+0000 7fd91b7fe700 1 -- 172.21.15.173:0/1840653932 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd9140136c0 con 0x7fd924060280 2024-01-21T19:51:21.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.809+0000 7fd9292f5700 1 -- 172.21.15.173:0/1840653932 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd924106db0 con 0x7fd924060280 2024-01-21T19:51:21.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.809+0000 7fd91b7fe700 1 -- 172.21.15.173:0/1840653932 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd91401ee90 con 0x7fd924060280 2024-01-21T19:51:21.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.809+0000 7fd9292f5700 1 -- 172.21.15.173:0/1840653932 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd924107240 con 0x7fd924060280 2024-01-21T19:51:21.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.810+0000 7fd91b7fe700 1 -- 172.21.15.173:0/1840653932 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd91401e420 con 0x7fd924060280 2024-01-21T19:51:21.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.810+0000 7fd9292f5700 1 -- 172.21.15.173:0/1840653932 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd924045ad0 con 0x7fd924060280 2024-01-21T19:51:21.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.810+0000 7fd91b7fe700 1 --2- 172.21.15.173:0/1840653932 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd91002ff10 0x7fd9100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:21.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.810+0000 7fd91b7fe700 1 -- 172.21.15.173:0/1840653932 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd914040f30 con 0x7fd924060280 2024-01-21T19:51:21.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.810+0000 7fd9227fc700 1 --2- 172.21.15.173:0/1840653932 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd91002ff10 0x7fd9100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:21.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.811+0000 7fd9227fc700 1 --2- 172.21.15.173:0/1840653932 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd91002ff10 0x7fd9100323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fd90c0096a0 tx=0x7fd90c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:21.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:21.815+0000 7fd91b7fe700 1 -- 172.21.15.173:0/1840653932 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd91406d2b0 con 0x7fd924060280 2024-01-21T19:51:22.010 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:22.018+0000 7fd9292f5700 1 -- 172.21.15.173:0/1840653932 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9241080f0 con 0x7fd924060280 2024-01-21T19:51:22.011 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:22.019+0000 7fd91b7fe700 1 -- 172.21.15.173:0/1840653932 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd91401c790 con 0x7fd924060280 2024-01-21T19:51:22.011 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:51:22.011 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:51:22.013 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:22.020+0000 7fd9292f5700 1 -- 172.21.15.173:0/1840653932 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd91002ff10 msgr2=0x7fd9100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:22.013 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:22.020+0000 7fd9292f5700 1 --2- 172.21.15.173:0/1840653932 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd91002ff10 0x7fd9100323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fd90c0096a0 tx=0x7fd90c006b40).stop 2024-01-21T19:51:22.013 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:22.020+0000 7fd9292f5700 1 -- 172.21.15.173:0/1840653932 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd924060280 msgr2=0x7fd9241066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:22.013 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:22.020+0000 7fd9292f5700 1 --2- 172.21.15.173:0/1840653932 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd924060280 0x7fd9241066c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fd914009830 tx=0x7fd91400b040).stop 2024-01-21T19:51:22.013 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:22.020+0000 7fd9292f5700 1 -- 172.21.15.173:0/1840653932 shutdown_connections 2024-01-21T19:51:22.013 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:22.020+0000 7fd9292f5700 1 --2- 172.21.15.173:0/1840653932 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd91002ff10 0x7fd9100323d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:22.013 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:22.020+0000 7fd9292f5700 1 --2- 172.21.15.173:0/1840653932 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd924060280 0x7fd9241066c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:22.013 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:22.020+0000 7fd9292f5700 1 -- 172.21.15.173:0/1840653932 >> 172.21.15.173:0/1840653932 conn(0x7fd9240f0b70 msgr2=0x7fd9240f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:22.013 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:22.021+0000 7fd9292f5700 1 -- 172.21.15.173:0/1840653932 shutdown_connections 2024-01-21T19:51:22.013 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:22.021+0000 7fd9292f5700 1 -- 172.21.15.173:0/1840653932 wait complete. 2024-01-21T19:51:22.014 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:23.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:22 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:20.958916+0000 mgr.smithi169.espgue (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:23.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:22 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:22.019683+0000 mon.smithi169 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.173:0/1840653932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:51:23.411 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:51:23.411 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:51:24.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.612+0000 7fb08a058700 1 -- 172.21.15.173:0/2808356742 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0840f4b80 msgr2=0x7fb0840f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:24.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.612+0000 7fb08a058700 1 --2- 172.21.15.173:0/2808356742 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0840f4b80 0x7fb0840f4fa0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fb074009fb0 tx=0x7fb074009b30).stop 2024-01-21T19:51:24.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.612+0000 7fb08a058700 1 -- 172.21.15.173:0/2808356742 shutdown_connections 2024-01-21T19:51:24.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.612+0000 7fb08a058700 1 --2- 172.21.15.173:0/2808356742 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0840f4b80 0x7fb0840f4fa0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:24.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.612+0000 7fb08a058700 1 -- 172.21.15.173:0/2808356742 >> 172.21.15.173:0/2808356742 conn(0x7fb0840f02e0 msgr2=0x7fb0840f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:24.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.612+0000 7fb08a058700 1 -- 172.21.15.173:0/2808356742 shutdown_connections 2024-01-21T19:51:24.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.612+0000 7fb08a058700 1 -- 172.21.15.173:0/2808356742 wait complete. 2024-01-21T19:51:24.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.613+0000 7fb08a058700 1 Processor -- start 2024-01-21T19:51:24.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.613+0000 7fb08a058700 1 -- start start 2024-01-21T19:51:24.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.613+0000 7fb08a058700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0840f4b80 0x7fb084109410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:24.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.613+0000 7fb08a058700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0840f18f0 con 0x7fb0840f4b80 2024-01-21T19:51:24.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.614+0000 7fb0837fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0840f4b80 0x7fb084109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:24.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.614+0000 7fb0837fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0840f4b80 0x7fb084109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:34692/0 (socket says 172.21.15.173:34692) 2024-01-21T19:51:24.608 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.614+0000 7fb0837fe700 1 -- 172.21.15.173:0/3314477745 learned_addr learned my addr 172.21.15.173:0/3314477745 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:51:24.608 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.614+0000 7fb0837fe700 1 -- 172.21.15.173:0/3314477745 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb074005040 con 0x7fb0840f4b80 2024-01-21T19:51:24.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.614+0000 7fb0837fe700 1 --2- 172.21.15.173:0/3314477745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0840f4b80 0x7fb084109410 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fb074013510 tx=0x7fb07400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:24.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.614+0000 7fb06bfff700 1 -- 172.21.15.173:0/3314477745 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb074009320 con 0x7fb0840f4b80 2024-01-21T19:51:24.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.614+0000 7fb06bfff700 1 -- 172.21.15.173:0/3314477745 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb074013f00 con 0x7fb0840f4b80 2024-01-21T19:51:24.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.614+0000 7fb08a058700 1 -- 172.21.15.173:0/3314477745 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0840f1af0 con 0x7fb0840f4b80 2024-01-21T19:51:24.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.614+0000 7fb06bfff700 1 -- 172.21.15.173:0/3314477745 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb074014d90 con 0x7fb0840f4b80 2024-01-21T19:51:24.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.614+0000 7fb08a058700 1 -- 172.21.15.173:0/3314477745 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb084109ac0 con 0x7fb0840f4b80 2024-01-21T19:51:24.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.615+0000 7fb06bfff700 1 -- 172.21.15.173:0/3314477745 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb074014420 con 0x7fb0840f4b80 2024-01-21T19:51:24.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.615+0000 7fb08a058700 1 -- 172.21.15.173:0/3314477745 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb084059100 con 0x7fb0840f4b80 2024-01-21T19:51:24.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.615+0000 7fb06bfff700 1 --2- 172.21.15.173:0/3314477745 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb06c02ff60 0x7fb06c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:24.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.615+0000 7fb06bfff700 1 -- 172.21.15.173:0/3314477745 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb074040fb0 con 0x7fb0840f4b80 2024-01-21T19:51:24.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.615+0000 7fb082ffd700 1 --2- 172.21.15.173:0/3314477745 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb06c02ff60 0x7fb06c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:24.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.616+0000 7fb082ffd700 1 --2- 172.21.15.173:0/3314477745 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb06c02ff60 0x7fb06c032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fb0780096a0 tx=0x7fb078006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:24.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.620+0000 7fb06bfff700 1 -- 172.21.15.173:0/3314477745 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb0740095f0 con 0x7fb0840f4b80 2024-01-21T19:51:24.818 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.825+0000 7fb08a058700 1 -- 172.21.15.173:0/3314477745 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb084045ad0 con 0x7fb0840f4b80 2024-01-21T19:51:24.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.826+0000 7fb06bfff700 1 -- 172.21.15.173:0/3314477745 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb07401db70 con 0x7fb0840f4b80 2024-01-21T19:51:24.819 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:51:24.819 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:51:24.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.828+0000 7fb08a058700 1 -- 172.21.15.173:0/3314477745 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb06c02ff60 msgr2=0x7fb06c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:24.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.828+0000 7fb08a058700 1 --2- 172.21.15.173:0/3314477745 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb06c02ff60 0x7fb06c032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fb0780096a0 tx=0x7fb078006b40).stop 2024-01-21T19:51:24.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.828+0000 7fb08a058700 1 -- 172.21.15.173:0/3314477745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0840f4b80 msgr2=0x7fb084109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:24.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.828+0000 7fb08a058700 1 --2- 172.21.15.173:0/3314477745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0840f4b80 0x7fb084109410 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fb074013510 tx=0x7fb07400b040).stop 2024-01-21T19:51:24.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.828+0000 7fb08a058700 1 -- 172.21.15.173:0/3314477745 shutdown_connections 2024-01-21T19:51:24.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.828+0000 7fb08a058700 1 --2- 172.21.15.173:0/3314477745 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb06c02ff60 0x7fb06c032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:24.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.828+0000 7fb08a058700 1 --2- 172.21.15.173:0/3314477745 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0840f4b80 0x7fb084109410 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:24.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.828+0000 7fb08a058700 1 -- 172.21.15.173:0/3314477745 >> 172.21.15.173:0/3314477745 conn(0x7fb0840f02e0 msgr2=0x7fb0840f2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:24.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.828+0000 7fb08a058700 1 -- 172.21.15.173:0/3314477745 shutdown_connections 2024-01-21T19:51:24.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:24.828+0000 7fb08a058700 1 -- 172.21.15.173:0/3314477745 wait complete. 2024-01-21T19:51:24.822 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:25.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:24 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:22.959261+0000 mgr.smithi169.espgue (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:25.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:24 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:24.827527+0000 mon.smithi169 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.173:0/3314477745' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:51:26.214 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:51:26.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:51:27.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:26 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:24.959590+0000 mgr.smithi169.espgue (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:27.462 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.469+0000 7fdaa2ba8700 1 -- 172.21.15.173:0/4162506931 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda9c0f6510 msgr2=0x7fda9c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:27.462 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.469+0000 7fdaa2ba8700 1 --2- 172.21.15.173:0/4162506931 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda9c0f6510 0x7fda9c0f6930 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fda90009fb0 tx=0x7fda90009b30).stop 2024-01-21T19:51:27.462 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.469+0000 7fdaa2ba8700 1 -- 172.21.15.173:0/4162506931 shutdown_connections 2024-01-21T19:51:27.462 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.469+0000 7fdaa2ba8700 1 --2- 172.21.15.173:0/4162506931 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda9c0f6510 0x7fda9c0f6930 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:27.463 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.469+0000 7fdaa2ba8700 1 -- 172.21.15.173:0/4162506931 >> 172.21.15.173:0/4162506931 conn(0x7fda9c0f1c70 msgr2=0x7fda9c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:27.463 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.469+0000 7fdaa2ba8700 1 -- 172.21.15.173:0/4162506931 shutdown_connections 2024-01-21T19:51:27.463 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.469+0000 7fdaa2ba8700 1 -- 172.21.15.173:0/4162506931 wait complete. 2024-01-21T19:51:27.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.469+0000 7fdaa2ba8700 1 Processor -- start 2024-01-21T19:51:27.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.470+0000 7fdaa2ba8700 1 -- start start 2024-01-21T19:51:27.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.470+0000 7fdaa2ba8700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda9c0f6510 0x7fda9c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:27.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.470+0000 7fdaa2ba8700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9c106c20 con 0x7fda9c0f6510 2024-01-21T19:51:27.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.470+0000 7fdaa0944700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda9c0f6510 0x7fda9c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:27.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.470+0000 7fdaa0944700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda9c0f6510 0x7fda9c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:34706/0 (socket says 172.21.15.173:34706) 2024-01-21T19:51:27.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.470+0000 7fdaa0944700 1 -- 172.21.15.173:0/1535220292 learned_addr learned my addr 172.21.15.173:0/1535220292 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:51:27.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.470+0000 7fdaa0944700 1 -- 172.21.15.173:0/1535220292 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda90005040 con 0x7fda9c0f6510 2024-01-21T19:51:27.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.471+0000 7fdaa0944700 1 --2- 172.21.15.173:0/1535220292 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda9c0f6510 0x7fda9c106730 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fda90013510 tx=0x7fda9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:27.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.471+0000 7fda997fa700 1 -- 172.21.15.173:0/1535220292 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda90009320 con 0x7fda9c0f6510 2024-01-21T19:51:27.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.471+0000 7fdaa2ba8700 1 -- 172.21.15.173:0/1535220292 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda9c106e20 con 0x7fda9c0f6510 2024-01-21T19:51:27.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.471+0000 7fda997fa700 1 -- 172.21.15.173:0/1535220292 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda90013f00 con 0x7fda9c0f6510 2024-01-21T19:51:27.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.471+0000 7fda997fa700 1 -- 172.21.15.173:0/1535220292 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda90014d90 con 0x7fda9c0f6510 2024-01-21T19:51:27.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.471+0000 7fdaa2ba8700 1 -- 172.21.15.173:0/1535220292 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda9c1072b0 con 0x7fda9c0f6510 2024-01-21T19:51:27.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.471+0000 7fda997fa700 1 -- 172.21.15.173:0/1535220292 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fda900095f0 con 0x7fda9c0f6510 2024-01-21T19:51:27.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.472+0000 7fdaa2ba8700 1 -- 172.21.15.173:0/1535220292 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda9c045ad0 con 0x7fda9c0f6510 2024-01-21T19:51:27.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.472+0000 7fda997fa700 1 --2- 172.21.15.173:0/1535220292 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fda8c02ff60 0x7fda8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:27.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.472+0000 7fda997fa700 1 -- 172.21.15.173:0/1535220292 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fda90040fe0 con 0x7fda9c0f6510 2024-01-21T19:51:27.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.472+0000 7fda9bfff700 1 --2- 172.21.15.173:0/1535220292 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fda8c02ff60 0x7fda8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:27.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.472+0000 7fda9bfff700 1 --2- 172.21.15.173:0/1535220292 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fda8c02ff60 0x7fda8c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fda880096a0 tx=0x7fda88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:27.469 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.476+0000 7fda997fa700 1 -- 172.21.15.173:0/1535220292 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda9001d7f0 con 0x7fda9c0f6510 2024-01-21T19:51:27.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.681+0000 7fdaa2ba8700 1 -- 172.21.15.173:0/1535220292 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda9c0fbf60 con 0x7fda9c0f6510 2024-01-21T19:51:27.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.682+0000 7fda997fa700 1 -- 172.21.15.173:0/1535220292 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fda90010230 con 0x7fda9c0f6510 2024-01-21T19:51:27.675 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:51:27.675 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:51:27.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.683+0000 7fdaa2ba8700 1 -- 172.21.15.173:0/1535220292 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fda8c02ff60 msgr2=0x7fda8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:27.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.683+0000 7fdaa2ba8700 1 --2- 172.21.15.173:0/1535220292 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fda8c02ff60 0x7fda8c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fda880096a0 tx=0x7fda88006b40).stop 2024-01-21T19:51:27.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.683+0000 7fdaa2ba8700 1 -- 172.21.15.173:0/1535220292 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda9c0f6510 msgr2=0x7fda9c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:27.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.683+0000 7fdaa2ba8700 1 --2- 172.21.15.173:0/1535220292 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda9c0f6510 0x7fda9c106730 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fda90013510 tx=0x7fda9000b040).stop 2024-01-21T19:51:27.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.683+0000 7fdaa2ba8700 1 -- 172.21.15.173:0/1535220292 shutdown_connections 2024-01-21T19:51:27.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.683+0000 7fdaa2ba8700 1 --2- 172.21.15.173:0/1535220292 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fda8c02ff60 0x7fda8c032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:27.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.683+0000 7fdaa2ba8700 1 --2- 172.21.15.173:0/1535220292 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fda9c0f6510 0x7fda9c106730 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:27.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.683+0000 7fdaa2ba8700 1 -- 172.21.15.173:0/1535220292 >> 172.21.15.173:0/1535220292 conn(0x7fda9c0f1c70 msgr2=0x7fda9c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:27.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.683+0000 7fdaa2ba8700 1 -- 172.21.15.173:0/1535220292 shutdown_connections 2024-01-21T19:51:27.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:27.683+0000 7fdaa2ba8700 1 -- 172.21.15.173:0/1535220292 wait complete. 2024-01-21T19:51:27.677 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:28.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:28 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:26.959970+0000 mgr.smithi169.espgue (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:28.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:28 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:27.683200+0000 mon.smithi169 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.173:0/1535220292' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:51:29.072 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:51:29.073 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:51:30.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.315+0000 7fec4e1e9700 1 -- 172.21.15.173:0/484990771 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec480f62a0 msgr2=0x7fec480f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:30.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.315+0000 7fec4e1e9700 1 --2- 172.21.15.173:0/484990771 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec480f62a0 0x7fec480f66c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fec38009fb0 tx=0x7fec38009b30).stop 2024-01-21T19:51:30.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.315+0000 7fec4e1e9700 1 -- 172.21.15.173:0/484990771 shutdown_connections 2024-01-21T19:51:30.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.315+0000 7fec4e1e9700 1 --2- 172.21.15.173:0/484990771 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec480f62a0 0x7fec480f66c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:30.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.315+0000 7fec4e1e9700 1 -- 172.21.15.173:0/484990771 >> 172.21.15.173:0/484990771 conn(0x7fec480f1aa0 msgr2=0x7fec480f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:30.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.315+0000 7fec4e1e9700 1 -- 172.21.15.173:0/484990771 shutdown_connections 2024-01-21T19:51:30.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.315+0000 7fec4e1e9700 1 -- 172.21.15.173:0/484990771 wait complete. 2024-01-21T19:51:30.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.315+0000 7fec4e1e9700 1 Processor -- start 2024-01-21T19:51:30.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.315+0000 7fec4e1e9700 1 -- start start 2024-01-21T19:51:30.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.316+0000 7fec4e1e9700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec480f62a0 0x7fec48108700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:30.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.316+0000 7fec4e1e9700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec48108bf0 con 0x7fec480f62a0 2024-01-21T19:51:30.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.316+0000 7fec4d1e7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec480f62a0 0x7fec48108700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:30.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.316+0000 7fec4d1e7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec480f62a0 0x7fec48108700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:34726/0 (socket says 172.21.15.173:34726) 2024-01-21T19:51:30.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.316+0000 7fec4d1e7700 1 -- 172.21.15.173:0/3816779976 learned_addr learned my addr 172.21.15.173:0/3816779976 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:51:30.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.316+0000 7fec4d1e7700 1 -- 172.21.15.173:0/3816779976 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec38005040 con 0x7fec480f62a0 2024-01-21T19:51:30.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.316+0000 7fec4d1e7700 1 --2- 172.21.15.173:0/3816779976 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec480f62a0 0x7fec48108700 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fec38013510 tx=0x7fec3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:30.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.316+0000 7fec3dffb700 1 -- 172.21.15.173:0/3816779976 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec38009320 con 0x7fec480f62a0 2024-01-21T19:51:30.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.316+0000 7fec4e1e9700 1 -- 172.21.15.173:0/3816779976 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec48108df0 con 0x7fec480f62a0 2024-01-21T19:51:30.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.316+0000 7fec3dffb700 1 -- 172.21.15.173:0/3816779976 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec38013f00 con 0x7fec480f62a0 2024-01-21T19:51:30.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.316+0000 7fec3dffb700 1 -- 172.21.15.173:0/3816779976 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec38014dc0 con 0x7fec480f62a0 2024-01-21T19:51:30.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.317+0000 7fec4e1e9700 1 -- 172.21.15.173:0/3816779976 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec48109280 con 0x7fec480f62a0 2024-01-21T19:51:30.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.317+0000 7fec3dffb700 1 -- 172.21.15.173:0/3816779976 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec3801d3d0 con 0x7fec480f62a0 2024-01-21T19:51:30.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.317+0000 7fec3dffb700 1 --2- 172.21.15.173:0/3816779976 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fec3402ff60 0x7fec34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:30.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.317+0000 7fec3dffb700 1 -- 172.21.15.173:0/3816779976 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec38041320 con 0x7fec480f62a0 2024-01-21T19:51:30.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.317+0000 7fec4e1e9700 1 -- 172.21.15.173:0/3816779976 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec48045ad0 con 0x7fec480f62a0 2024-01-21T19:51:30.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.318+0000 7fec4c9e6700 1 --2- 172.21.15.173:0/3816779976 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fec3402ff60 0x7fec34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:30.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.318+0000 7fec4c9e6700 1 --2- 172.21.15.173:0/3816779976 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fec3402ff60 0x7fec34032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fec440096a0 tx=0x7fec44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:30.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.322+0000 7fec3dffb700 1 -- 172.21.15.173:0/3816779976 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec380263a0 con 0x7fec480f62a0 2024-01-21T19:51:30.530 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.537+0000 7fec4e1e9700 1 -- 172.21.15.173:0/3816779976 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec4810a790 con 0x7fec480f62a0 2024-01-21T19:51:30.530 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.537+0000 7fec3dffb700 1 -- 172.21.15.173:0/3816779976 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fec3802b030 con 0x7fec480f62a0 2024-01-21T19:51:30.531 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:51:30.531 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:51:30.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.539+0000 7fec4e1e9700 1 -- 172.21.15.173:0/3816779976 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fec3402ff60 msgr2=0x7fec34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:30.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.539+0000 7fec4e1e9700 1 --2- 172.21.15.173:0/3816779976 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fec3402ff60 0x7fec34032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fec440096a0 tx=0x7fec44006b40).stop 2024-01-21T19:51:30.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.539+0000 7fec4e1e9700 1 -- 172.21.15.173:0/3816779976 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec480f62a0 msgr2=0x7fec48108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:30.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.539+0000 7fec4e1e9700 1 --2- 172.21.15.173:0/3816779976 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec480f62a0 0x7fec48108700 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fec38013510 tx=0x7fec3800b040).stop 2024-01-21T19:51:30.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.539+0000 7fec4e1e9700 1 -- 172.21.15.173:0/3816779976 shutdown_connections 2024-01-21T19:51:30.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.539+0000 7fec4e1e9700 1 --2- 172.21.15.173:0/3816779976 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fec3402ff60 0x7fec34032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:30.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.539+0000 7fec4e1e9700 1 --2- 172.21.15.173:0/3816779976 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fec480f62a0 0x7fec48108700 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:30.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.539+0000 7fec4e1e9700 1 -- 172.21.15.173:0/3816779976 >> 172.21.15.173:0/3816779976 conn(0x7fec480f1aa0 msgr2=0x7fec480f2880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:30.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.539+0000 7fec4e1e9700 1 -- 172.21.15.173:0/3816779976 shutdown_connections 2024-01-21T19:51:30.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:30.539+0000 7fec4e1e9700 1 -- 172.21.15.173:0/3816779976 wait complete. 2024-01-21T19:51:30.533 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:30.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:30 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:28.960324+0000 mgr.smithi169.espgue (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:30.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:30 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:30.539068+0000 mon.smithi169 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.173:0/3816779976' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:51:31.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:31.071710+0000 mon.smithi169 (mon.0) 175 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/mirror_snapshot_schedule"}]: dispatch 2024-01-21T19:51:31.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:31.075415+0000 mon.smithi169 (mon.0) 176 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/trash_purge_schedule"}]: dispatch 2024-01-21T19:51:31.924 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:51:31.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:51:32.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:32 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:30.960594+0000 mgr.smithi169.espgue (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:33.131 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.137+0000 7f187f59e700 1 -- 172.21.15.173:0/3596728959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18800f62a0 msgr2=0x7f18800f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:33.131 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.137+0000 7f187f59e700 1 --2- 172.21.15.173:0/3596728959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18800f62a0 0x7f18800f66c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f1870009fb0 tx=0x7f1870009b30).stop 2024-01-21T19:51:33.131 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.137+0000 7f187f59e700 1 -- 172.21.15.173:0/3596728959 shutdown_connections 2024-01-21T19:51:33.131 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.137+0000 7f187f59e700 1 --2- 172.21.15.173:0/3596728959 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18800f62a0 0x7f18800f66c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:33.131 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.137+0000 7f187f59e700 1 -- 172.21.15.173:0/3596728959 >> 172.21.15.173:0/3596728959 conn(0x7f18800f1aa0 msgr2=0x7f18800f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:33.131 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.137+0000 7f187f59e700 1 -- 172.21.15.173:0/3596728959 shutdown_connections 2024-01-21T19:51:33.131 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.137+0000 7f187f59e700 1 -- 172.21.15.173:0/3596728959 wait complete. 2024-01-21T19:51:33.131 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.137+0000 7f187f59e700 1 Processor -- start 2024-01-21T19:51:33.131 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.137+0000 7f187f59e700 1 -- start start 2024-01-21T19:51:33.131 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.137+0000 7f187f59e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18800f62a0 0x7f18801064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:33.132 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.137+0000 7f187f59e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18801069a0 con 0x7f18800f62a0 2024-01-21T19:51:33.132 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.138+0000 7f187e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18800f62a0 0x7f18801064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:33.132 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.138+0000 7f187e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18800f62a0 0x7f18801064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:34742/0 (socket says 172.21.15.173:34742) 2024-01-21T19:51:33.132 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.138+0000 7f187e59c700 1 -- 172.21.15.173:0/2705668354 learned_addr learned my addr 172.21.15.173:0/2705668354 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:51:33.133 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.138+0000 7f187e59c700 1 -- 172.21.15.173:0/2705668354 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1870005040 con 0x7f18800f62a0 2024-01-21T19:51:33.134 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.138+0000 7f187e59c700 1 --2- 172.21.15.173:0/2705668354 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18800f62a0 0x7f18801064b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f1870013510 tx=0x7f187000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:33.134 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.138+0000 7f186effd700 1 -- 172.21.15.173:0/2705668354 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1870009320 con 0x7f18800f62a0 2024-01-21T19:51:33.134 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.138+0000 7f187f59e700 1 -- 172.21.15.173:0/2705668354 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1880106ba0 con 0x7f18800f62a0 2024-01-21T19:51:33.134 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.138+0000 7f186effd700 1 -- 172.21.15.173:0/2705668354 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1870013f00 con 0x7f18800f62a0 2024-01-21T19:51:33.134 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.138+0000 7f186effd700 1 -- 172.21.15.173:0/2705668354 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1870014dc0 con 0x7f18800f62a0 2024-01-21T19:51:33.134 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.138+0000 7f187f59e700 1 -- 172.21.15.173:0/2705668354 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1880107030 con 0x7f18800f62a0 2024-01-21T19:51:33.134 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.139+0000 7f186effd700 1 -- 172.21.15.173:0/2705668354 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f187001d3d0 con 0x7f18800f62a0 2024-01-21T19:51:33.134 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.139+0000 7f186effd700 1 --2- 172.21.15.173:0/2705668354 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f186802ffb0 0x7f1868032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:33.134 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.139+0000 7f186effd700 1 -- 172.21.15.173:0/2705668354 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f18700095f0 con 0x7f18800f62a0 2024-01-21T19:51:33.134 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.139+0000 7f187f59e700 1 -- 172.21.15.173:0/2705668354 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1880059100 con 0x7f18800f62a0 2024-01-21T19:51:33.134 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.140+0000 7f187dd9b700 1 --2- 172.21.15.173:0/2705668354 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f186802ffb0 0x7f1868032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:33.134 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.140+0000 7f187dd9b700 1 --2- 172.21.15.173:0/2705668354 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f186802ffb0 0x7f1868032470 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f18740096a0 tx=0x7f1874006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:33.137 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.144+0000 7f186effd700 1 -- 172.21.15.173:0/2705668354 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1870012920 con 0x7f18800f62a0 2024-01-21T19:51:33.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.349+0000 7f187f59e700 1 -- 172.21.15.173:0/2705668354 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18800fbce0 con 0x7f18800f62a0 2024-01-21T19:51:33.343 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.350+0000 7f186effd700 1 -- 172.21.15.173:0/2705668354 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f187002b020 con 0x7f18800f62a0 2024-01-21T19:51:33.343 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:51:33.343 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:51:33.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.351+0000 7f187f59e700 1 -- 172.21.15.173:0/2705668354 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f186802ffb0 msgr2=0x7f1868032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:33.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.351+0000 7f187f59e700 1 --2- 172.21.15.173:0/2705668354 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f186802ffb0 0x7f1868032470 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f18740096a0 tx=0x7f1874006b40).stop 2024-01-21T19:51:33.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.351+0000 7f187f59e700 1 -- 172.21.15.173:0/2705668354 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18800f62a0 msgr2=0x7f18801064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:33.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.351+0000 7f187f59e700 1 --2- 172.21.15.173:0/2705668354 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18800f62a0 0x7f18801064b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f1870013510 tx=0x7f187000b040).stop 2024-01-21T19:51:33.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.351+0000 7f187f59e700 1 -- 172.21.15.173:0/2705668354 shutdown_connections 2024-01-21T19:51:33.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.351+0000 7f187f59e700 1 --2- 172.21.15.173:0/2705668354 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f186802ffb0 0x7f1868032470 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:33.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.351+0000 7f187f59e700 1 --2- 172.21.15.173:0/2705668354 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18800f62a0 0x7f18801064b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:33.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.351+0000 7f187f59e700 1 -- 172.21.15.173:0/2705668354 >> 172.21.15.173:0/2705668354 conn(0x7f18800f1aa0 msgr2=0x7f18800f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:33.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.351+0000 7f187f59e700 1 -- 172.21.15.173:0/2705668354 shutdown_connections 2024-01-21T19:51:33.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:33.351+0000 7f187f59e700 1 -- 172.21.15.173:0/2705668354 wait complete. 2024-01-21T19:51:33.345 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:33.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:33 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:33.351407+0000 mon.smithi169 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.173:0/2705668354' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:51:34.735 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:51:34.735 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:51:34.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:34 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:32.960863+0000 mgr.smithi169.espgue (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:35.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.911+0000 7f7f89451700 1 -- 172.21.15.173:0/618757805 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f840f6510 msgr2=0x7f7f840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:35.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.911+0000 7f7f89451700 1 --2- 172.21.15.173:0/618757805 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f840f6510 0x7f7f840f6930 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f7f74009fb0 tx=0x7f7f74009b30).stop 2024-01-21T19:51:35.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.911+0000 7f7f89451700 1 -- 172.21.15.173:0/618757805 shutdown_connections 2024-01-21T19:51:35.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.911+0000 7f7f89451700 1 --2- 172.21.15.173:0/618757805 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f840f6510 0x7f7f840f6930 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:35.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.911+0000 7f7f89451700 1 -- 172.21.15.173:0/618757805 >> 172.21.15.173:0/618757805 conn(0x7f7f840f1c70 msgr2=0x7f7f840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:35.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.911+0000 7f7f89451700 1 -- 172.21.15.173:0/618757805 shutdown_connections 2024-01-21T19:51:35.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.911+0000 7f7f89451700 1 -- 172.21.15.173:0/618757805 wait complete. 2024-01-21T19:51:35.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.912+0000 7f7f89451700 1 Processor -- start 2024-01-21T19:51:35.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.912+0000 7f7f89451700 1 -- start start 2024-01-21T19:51:35.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.912+0000 7f7f89451700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f840f6510 0x7f7f84106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:35.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.912+0000 7f7f89451700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f84106c20 con 0x7f7f840f6510 2024-01-21T19:51:35.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.912+0000 7f7f82ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f840f6510 0x7f7f84106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:35.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.912+0000 7f7f82ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f840f6510 0x7f7f84106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:45858/0 (socket says 172.21.15.173:45858) 2024-01-21T19:51:35.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.912+0000 7f7f82ffd700 1 -- 172.21.15.173:0/62796785 learned_addr learned my addr 172.21.15.173:0/62796785 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:51:35.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.912+0000 7f7f82ffd700 1 -- 172.21.15.173:0/62796785 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f74005040 con 0x7f7f840f6510 2024-01-21T19:51:35.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.912+0000 7f7f82ffd700 1 --2- 172.21.15.173:0/62796785 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f840f6510 0x7f7f84106730 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f7f74013510 tx=0x7f7f7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:35.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.913+0000 7f7f6b7fe700 1 -- 172.21.15.173:0/62796785 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f74009320 con 0x7f7f840f6510 2024-01-21T19:51:35.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.913+0000 7f7f89451700 1 -- 172.21.15.173:0/62796785 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f84106e20 con 0x7f7f840f6510 2024-01-21T19:51:35.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.913+0000 7f7f6b7fe700 1 -- 172.21.15.173:0/62796785 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f74013f00 con 0x7f7f840f6510 2024-01-21T19:51:35.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.913+0000 7f7f6b7fe700 1 -- 172.21.15.173:0/62796785 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f74014d90 con 0x7f7f840f6510 2024-01-21T19:51:35.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.913+0000 7f7f89451700 1 -- 172.21.15.173:0/62796785 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f841072b0 con 0x7f7f840f6510 2024-01-21T19:51:35.908 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.913+0000 7f7f6b7fe700 1 -- 172.21.15.173:0/62796785 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f74014420 con 0x7f7f840f6510 2024-01-21T19:51:35.908 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.914+0000 7f7f6b7fe700 1 --2- 172.21.15.173:0/62796785 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f7f6c02ff60 0x7f7f6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:35.908 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.914+0000 7f7f89451700 1 -- 172.21.15.173:0/62796785 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f84045ad0 con 0x7f7f840f6510 2024-01-21T19:51:35.908 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.914+0000 7f7f6b7fe700 1 -- 172.21.15.173:0/62796785 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f74041000 con 0x7f7f840f6510 2024-01-21T19:51:35.908 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.914+0000 7f7f827fc700 1 --2- 172.21.15.173:0/62796785 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f7f6c02ff60 0x7f7f6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:35.908 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.914+0000 7f7f827fc700 1 --2- 172.21.15.173:0/62796785 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f7f6c02ff60 0x7f7f6c032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7f780096a0 tx=0x7f7f78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:35.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:35.918+0000 7f7f6b7fe700 1 -- 172.21.15.173:0/62796785 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f740095f0 con 0x7f7f840f6510 2024-01-21T19:51:36.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:36.122+0000 7f7f89451700 1 -- 172.21.15.173:0/62796785 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f840fc050 con 0x7f7f840f6510 2024-01-21T19:51:36.116 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:36.123+0000 7f7f6b7fe700 1 -- 172.21.15.173:0/62796785 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7f7402c090 con 0x7f7f840f6510 2024-01-21T19:51:36.116 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:51:36.116 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:51:36.118 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:36.125+0000 7f7f89451700 1 -- 172.21.15.173:0/62796785 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f7f6c02ff60 msgr2=0x7f7f6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:36.118 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:36.125+0000 7f7f89451700 1 --2- 172.21.15.173:0/62796785 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f7f6c02ff60 0x7f7f6c032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7f780096a0 tx=0x7f7f78006b40).stop 2024-01-21T19:51:36.118 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:36.125+0000 7f7f89451700 1 -- 172.21.15.173:0/62796785 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f840f6510 msgr2=0x7f7f84106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:36.118 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:36.125+0000 7f7f89451700 1 --2- 172.21.15.173:0/62796785 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f840f6510 0x7f7f84106730 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f7f74013510 tx=0x7f7f7400b040).stop 2024-01-21T19:51:36.118 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:36.125+0000 7f7f89451700 1 -- 172.21.15.173:0/62796785 shutdown_connections 2024-01-21T19:51:36.118 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:36.125+0000 7f7f89451700 1 --2- 172.21.15.173:0/62796785 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f7f6c02ff60 0x7f7f6c032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:36.119 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:36.125+0000 7f7f89451700 1 --2- 172.21.15.173:0/62796785 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7f840f6510 0x7f7f84106730 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:36.119 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:36.125+0000 7f7f89451700 1 -- 172.21.15.173:0/62796785 >> 172.21.15.173:0/62796785 conn(0x7f7f840f1c70 msgr2=0x7f7f840f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:36.119 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:36.125+0000 7f7f89451700 1 -- 172.21.15.173:0/62796785 shutdown_connections 2024-01-21T19:51:36.119 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:36.125+0000 7f7f89451700 1 -- 172.21.15.173:0/62796785 wait complete. 2024-01-21T19:51:36.119 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:36.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:36 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:34.961200+0000 mgr.smithi169.espgue (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:36.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:36 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:36.124591+0000 mon.smithi169 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.173:0/62796785' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:51:37.513 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:51:37.513 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:51:38.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.738+0000 7f9724d14700 1 -- 172.21.15.173:0/760445147 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97200f62a0 msgr2=0x7f97200f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:38.734 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.738+0000 7f9724d14700 1 --2- 172.21.15.173:0/760445147 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97200f62a0 0x7f97200f66c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f9710009fb0 tx=0x7f9710009b30).stop 2024-01-21T19:51:38.734 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.738+0000 7f9724d14700 1 -- 172.21.15.173:0/760445147 shutdown_connections 2024-01-21T19:51:38.734 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.738+0000 7f9724d14700 1 --2- 172.21.15.173:0/760445147 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97200f62a0 0x7f97200f66c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:38.734 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.738+0000 7f9724d14700 1 -- 172.21.15.173:0/760445147 >> 172.21.15.173:0/760445147 conn(0x7f97200f1aa0 msgr2=0x7f97200f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:38.734 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.738+0000 7f9724d14700 1 -- 172.21.15.173:0/760445147 shutdown_connections 2024-01-21T19:51:38.734 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.739+0000 7f9724d14700 1 -- 172.21.15.173:0/760445147 wait complete. 2024-01-21T19:51:38.734 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.739+0000 7f9724d14700 1 Processor -- start 2024-01-21T19:51:38.734 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.739+0000 7f9724d14700 1 -- start start 2024-01-21T19:51:38.734 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.739+0000 7f9724d14700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97200f62a0 0x7f9720108700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:38.734 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.739+0000 7f9724d14700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9720108bf0 con 0x7f97200f62a0 2024-01-21T19:51:38.734 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.740+0000 7f971f7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97200f62a0 0x7f9720108700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:38.734 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.740+0000 7f971f7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97200f62a0 0x7f9720108700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:45874/0 (socket says 172.21.15.173:45874) 2024-01-21T19:51:38.734 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.740+0000 7f971f7fe700 1 -- 172.21.15.173:0/3194844106 learned_addr learned my addr 172.21.15.173:0/3194844106 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:51:38.734 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.740+0000 7f971f7fe700 1 -- 172.21.15.173:0/3194844106 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9710005040 con 0x7f97200f62a0 2024-01-21T19:51:38.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.740+0000 7f971f7fe700 1 --2- 172.21.15.173:0/3194844106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97200f62a0 0x7f9720108700 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f9710013510 tx=0x7f971000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:38.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.740+0000 7f9707fff700 1 -- 172.21.15.173:0/3194844106 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9710009320 con 0x7f97200f62a0 2024-01-21T19:51:38.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.740+0000 7f9724d14700 1 -- 172.21.15.173:0/3194844106 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9720108df0 con 0x7f97200f62a0 2024-01-21T19:51:38.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.740+0000 7f9707fff700 1 -- 172.21.15.173:0/3194844106 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9710013f00 con 0x7f97200f62a0 2024-01-21T19:51:38.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.740+0000 7f9707fff700 1 -- 172.21.15.173:0/3194844106 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9710014dc0 con 0x7f97200f62a0 2024-01-21T19:51:38.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.740+0000 7f9724d14700 1 -- 172.21.15.173:0/3194844106 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9720109280 con 0x7f97200f62a0 2024-01-21T19:51:38.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.741+0000 7f9707fff700 1 -- 172.21.15.173:0/3194844106 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f971001d3d0 con 0x7f97200f62a0 2024-01-21T19:51:38.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.741+0000 7f9707fff700 1 --2- 172.21.15.173:0/3194844106 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f970802ff60 0x7f9708032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:38.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.741+0000 7f9724d14700 1 -- 172.21.15.173:0/3194844106 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9720045ad0 con 0x7f97200f62a0 2024-01-21T19:51:38.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.741+0000 7f9707fff700 1 -- 172.21.15.173:0/3194844106 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9710041400 con 0x7f97200f62a0 2024-01-21T19:51:38.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.742+0000 7f971effd700 1 --2- 172.21.15.173:0/3194844106 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f970802ff60 0x7f9708032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:38.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.742+0000 7f971effd700 1 --2- 172.21.15.173:0/3194844106 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f970802ff60 0x7f9708032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f97140096a0 tx=0x7f9714006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:38.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.746+0000 7f9707fff700 1 -- 172.21.15.173:0/3194844106 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9710026490 con 0x7f97200f62a0 2024-01-21T19:51:38.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:36.961496+0000 mgr.smithi169.espgue (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:38.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.953+0000 7f9724d14700 1 -- 172.21.15.173:0/3194844106 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f972010a8a0 con 0x7f97200f62a0 2024-01-21T19:51:38.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.955+0000 7f9707fff700 1 -- 172.21.15.173:0/3194844106 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f971001ccf0 con 0x7f97200f62a0 2024-01-21T19:51:38.947 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:51:38.947 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:51:38.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.955+0000 7f9724d14700 1 -- 172.21.15.173:0/3194844106 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f970802ff60 msgr2=0x7f9708032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:38.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.955+0000 7f9724d14700 1 --2- 172.21.15.173:0/3194844106 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f970802ff60 0x7f9708032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f97140096a0 tx=0x7f9714006b40).stop 2024-01-21T19:51:38.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.955+0000 7f9724d14700 1 -- 172.21.15.173:0/3194844106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97200f62a0 msgr2=0x7f9720108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:38.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.955+0000 7f9724d14700 1 --2- 172.21.15.173:0/3194844106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97200f62a0 0x7f9720108700 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f9710013510 tx=0x7f971000b040).stop 2024-01-21T19:51:38.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.955+0000 7f9724d14700 1 -- 172.21.15.173:0/3194844106 shutdown_connections 2024-01-21T19:51:38.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.955+0000 7f9724d14700 1 --2- 172.21.15.173:0/3194844106 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f970802ff60 0x7f9708032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:38.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.955+0000 7f9724d14700 1 --2- 172.21.15.173:0/3194844106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97200f62a0 0x7f9720108700 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:38.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.955+0000 7f9724d14700 1 -- 172.21.15.173:0/3194844106 >> 172.21.15.173:0/3194844106 conn(0x7f97200f1aa0 msgr2=0x7f97200f2880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:38.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.955+0000 7f9724d14700 1 -- 172.21.15.173:0/3194844106 shutdown_connections 2024-01-21T19:51:38.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:38.956+0000 7f9724d14700 1 -- 172.21.15.173:0/3194844106 wait complete. 2024-01-21T19:51:38.949 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:39.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:39 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:38.955495+0000 mon.smithi169 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.173:0/3194844106' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:51:40.329 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:51:40.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:51:40.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:40 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:38.961783+0000 mgr.smithi169.espgue (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:41.583 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.589+0000 7fe8f6dfe700 1 -- 172.21.15.173:0/1685058632 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8f00f6510 msgr2=0x7fe8f00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:41.583 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.589+0000 7fe8f6dfe700 1 --2- 172.21.15.173:0/1685058632 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8f00f6510 0x7fe8f00f6930 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fe8e4009fb0 tx=0x7fe8e4009b30).stop 2024-01-21T19:51:41.583 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.589+0000 7fe8f6dfe700 1 -- 172.21.15.173:0/1685058632 shutdown_connections 2024-01-21T19:51:41.583 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.589+0000 7fe8f6dfe700 1 --2- 172.21.15.173:0/1685058632 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8f00f6510 0x7fe8f00f6930 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:41.584 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.589+0000 7fe8f6dfe700 1 -- 172.21.15.173:0/1685058632 >> 172.21.15.173:0/1685058632 conn(0x7fe8f00f1c70 msgr2=0x7fe8f00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:41.584 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.589+0000 7fe8f6dfe700 1 -- 172.21.15.173:0/1685058632 shutdown_connections 2024-01-21T19:51:41.584 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.589+0000 7fe8f6dfe700 1 -- 172.21.15.173:0/1685058632 wait complete. 2024-01-21T19:51:41.584 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.590+0000 7fe8f6dfe700 1 Processor -- start 2024-01-21T19:51:41.584 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.590+0000 7fe8f6dfe700 1 -- start start 2024-01-21T19:51:41.584 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.590+0000 7fe8f6dfe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8f00f6510 0x7fe8f0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:41.584 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.590+0000 7fe8f6dfe700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f0106c20 con 0x7fe8f00f6510 2024-01-21T19:51:41.584 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.590+0000 7fe8f4b9a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8f00f6510 0x7fe8f0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:41.584 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.590+0000 7fe8f4b9a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8f00f6510 0x7fe8f0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:45882/0 (socket says 172.21.15.173:45882) 2024-01-21T19:51:41.584 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.590+0000 7fe8f4b9a700 1 -- 172.21.15.173:0/4233565203 learned_addr learned my addr 172.21.15.173:0/4233565203 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:51:41.586 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.591+0000 7fe8f4b9a700 1 -- 172.21.15.173:0/4233565203 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e4005040 con 0x7fe8f00f6510 2024-01-21T19:51:41.586 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.591+0000 7fe8f4b9a700 1 --2- 172.21.15.173:0/4233565203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8f00f6510 0x7fe8f0106730 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fe8e4013510 tx=0x7fe8e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:41.586 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.591+0000 7fe8ed7fa700 1 -- 172.21.15.173:0/4233565203 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8e4009320 con 0x7fe8f00f6510 2024-01-21T19:51:41.586 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.591+0000 7fe8f6dfe700 1 -- 172.21.15.173:0/4233565203 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f0106e20 con 0x7fe8f00f6510 2024-01-21T19:51:41.586 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.591+0000 7fe8ed7fa700 1 -- 172.21.15.173:0/4233565203 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe8e4013f00 con 0x7fe8f00f6510 2024-01-21T19:51:41.586 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.591+0000 7fe8ed7fa700 1 -- 172.21.15.173:0/4233565203 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8e4014d90 con 0x7fe8f00f6510 2024-01-21T19:51:41.586 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.591+0000 7fe8f6dfe700 1 -- 172.21.15.173:0/4233565203 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f01072b0 con 0x7fe8f00f6510 2024-01-21T19:51:41.586 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.592+0000 7fe8ed7fa700 1 -- 172.21.15.173:0/4233565203 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8e4014420 con 0x7fe8f00f6510 2024-01-21T19:51:41.586 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.592+0000 7fe8f6dfe700 1 -- 172.21.15.173:0/4233565203 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f0045ad0 con 0x7fe8f00f6510 2024-01-21T19:51:41.586 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.592+0000 7fe8ed7fa700 1 --2- 172.21.15.173:0/4233565203 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe8e002ff60 0x7fe8e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:41.587 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.592+0000 7fe8ed7fa700 1 -- 172.21.15.173:0/4233565203 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe8e4041050 con 0x7fe8f00f6510 2024-01-21T19:51:41.587 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.592+0000 7fe8effff700 1 --2- 172.21.15.173:0/4233565203 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe8e002ff60 0x7fe8e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:41.587 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.593+0000 7fe8effff700 1 --2- 172.21.15.173:0/4233565203 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe8e002ff60 0x7fe8e0032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fe8dc0096a0 tx=0x7fe8dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:41.589 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.597+0000 7fe8ed7fa700 1 -- 172.21.15.173:0/4233565203 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe8e40095f0 con 0x7fe8f00f6510 2024-01-21T19:51:41.795 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.802+0000 7fe8f6dfe700 1 -- 172.21.15.173:0/4233565203 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8f00fbf60 con 0x7fe8f00f6510 2024-01-21T19:51:41.796 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.804+0000 7fe8ed7fa700 1 -- 172.21.15.173:0/4233565203 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe8e401db70 con 0x7fe8f00f6510 2024-01-21T19:51:41.796 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:51:41.796 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:51:41.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.804+0000 7fe8f6dfe700 1 -- 172.21.15.173:0/4233565203 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe8e002ff60 msgr2=0x7fe8e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:41.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.804+0000 7fe8f6dfe700 1 --2- 172.21.15.173:0/4233565203 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe8e002ff60 0x7fe8e0032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fe8dc0096a0 tx=0x7fe8dc006b40).stop 2024-01-21T19:51:41.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.804+0000 7fe8f6dfe700 1 -- 172.21.15.173:0/4233565203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8f00f6510 msgr2=0x7fe8f0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:41.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.804+0000 7fe8f6dfe700 1 --2- 172.21.15.173:0/4233565203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8f00f6510 0x7fe8f0106730 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fe8e4013510 tx=0x7fe8e400b040).stop 2024-01-21T19:51:41.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.804+0000 7fe8f6dfe700 1 -- 172.21.15.173:0/4233565203 shutdown_connections 2024-01-21T19:51:41.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.804+0000 7fe8f6dfe700 1 --2- 172.21.15.173:0/4233565203 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe8e002ff60 0x7fe8e0032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:41.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.804+0000 7fe8f6dfe700 1 --2- 172.21.15.173:0/4233565203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8f00f6510 0x7fe8f0106730 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:41.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.804+0000 7fe8f6dfe700 1 -- 172.21.15.173:0/4233565203 >> 172.21.15.173:0/4233565203 conn(0x7fe8f00f1c70 msgr2=0x7fe8f00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:41.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.804+0000 7fe8f6dfe700 1 -- 172.21.15.173:0/4233565203 shutdown_connections 2024-01-21T19:51:41.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:41.804+0000 7fe8f6dfe700 1 -- 172.21.15.173:0/4233565203 wait complete. 2024-01-21T19:51:41.798 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:43.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:40.962114+0000 mgr.smithi169.espgue (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:43.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:41.804358+0000 mon.smithi169 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.173:0/4233565203' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:51:43.214 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:51:43.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:51:44.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.436+0000 7f8b0e27d700 1 -- 172.21.15.173:0/581007823 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b080f51a0 msgr2=0x7f8b080f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:44.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.436+0000 7f8b0e27d700 1 --2- 172.21.15.173:0/581007823 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b080f51a0 0x7f8b080f55c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f8af8009fb0 tx=0x7f8af8009b30).stop 2024-01-21T19:51:44.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.436+0000 7f8b0e27d700 1 -- 172.21.15.173:0/581007823 shutdown_connections 2024-01-21T19:51:44.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.436+0000 7f8b0e27d700 1 --2- 172.21.15.173:0/581007823 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b080f51a0 0x7f8b080f55c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:44.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.436+0000 7f8b0e27d700 1 -- 172.21.15.173:0/581007823 >> 172.21.15.173:0/581007823 conn(0x7f8b080f09a0 msgr2=0x7f8b080f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:44.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.436+0000 7f8b0e27d700 1 -- 172.21.15.173:0/581007823 shutdown_connections 2024-01-21T19:51:44.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.436+0000 7f8b0e27d700 1 -- 172.21.15.173:0/581007823 wait complete. 2024-01-21T19:51:44.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.437+0000 7f8b0e27d700 1 Processor -- start 2024-01-21T19:51:44.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.437+0000 7f8b0e27d700 1 -- start start 2024-01-21T19:51:44.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.437+0000 7f8b0e27d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b080fd160 0x7f8b080fb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:44.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.437+0000 7f8b0e27d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b080fd580 con 0x7f8b080fd160 2024-01-21T19:51:44.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.437+0000 7f8b0d27b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b080fd160 0x7f8b080fb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:44.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.437+0000 7f8b0d27b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b080fd160 0x7f8b080fb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:56094/0 (socket says 172.21.15.173:56094) 2024-01-21T19:51:44.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.437+0000 7f8b0d27b700 1 -- 172.21.15.173:0/3832268737 learned_addr learned my addr 172.21.15.173:0/3832268737 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:51:44.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.438+0000 7f8b0d27b700 1 -- 172.21.15.173:0/3832268737 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af8005040 con 0x7f8b080fd160 2024-01-21T19:51:44.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.438+0000 7f8b0d27b700 1 --2- 172.21.15.173:0/3832268737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b080fd160 0x7f8b080fb620 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f8af8009f80 tx=0x7f8af8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:44.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.438+0000 7f8afdffb700 1 -- 172.21.15.173:0/3832268737 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8af80136c0 con 0x7f8b080fd160 2024-01-21T19:51:44.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.438+0000 7f8b0e27d700 1 -- 172.21.15.173:0/3832268737 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b080fbb10 con 0x7f8b080fd160 2024-01-21T19:51:44.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.438+0000 7f8afdffb700 1 -- 172.21.15.173:0/3832268737 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8af80095f0 con 0x7f8b080fd160 2024-01-21T19:51:44.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.438+0000 7f8afdffb700 1 -- 172.21.15.173:0/3832268737 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8af8014420 con 0x7f8b080fd160 2024-01-21T19:51:44.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.438+0000 7f8b0e27d700 1 -- 172.21.15.173:0/3832268737 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b080fc000 con 0x7f8b080fd160 2024-01-21T19:51:44.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.439+0000 7f8afdffb700 1 -- 172.21.15.173:0/3832268737 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8af8025060 con 0x7f8b080fd160 2024-01-21T19:51:44.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.439+0000 7f8b0e27d700 1 -- 172.21.15.173:0/3832268737 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b08059100 con 0x7f8b080fd160 2024-01-21T19:51:44.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.439+0000 7f8afdffb700 1 --2- 172.21.15.173:0/3832268737 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8af402ff60 0x7f8af4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:44.433 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.439+0000 7f8afdffb700 1 -- 172.21.15.173:0/3832268737 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8af8040da0 con 0x7f8b080fd160 2024-01-21T19:51:44.433 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.439+0000 7f8b0ca7a700 1 --2- 172.21.15.173:0/3832268737 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8af402ff60 0x7f8af4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:44.433 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.440+0000 7f8b0ca7a700 1 --2- 172.21.15.173:0/3832268737 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8af402ff60 0x7f8af4032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f8b040096a0 tx=0x7f8b04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:44.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.444+0000 7f8afdffb700 1 -- 172.21.15.173:0/3832268737 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8af801c950 con 0x7f8b080fd160 2024-01-21T19:51:44.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.648+0000 7f8b0e27d700 1 -- 172.21.15.173:0/3832268737 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8b080fc3f0 con 0x7f8b080fd160 2024-01-21T19:51:44.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.649+0000 7f8afdffb700 1 -- 172.21.15.173:0/3832268737 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8af801cb60 con 0x7f8b080fd160 2024-01-21T19:51:44.641 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:51:44.641 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:51:44.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.650+0000 7f8b0e27d700 1 -- 172.21.15.173:0/3832268737 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8af402ff60 msgr2=0x7f8af4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:44.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.650+0000 7f8b0e27d700 1 --2- 172.21.15.173:0/3832268737 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8af402ff60 0x7f8af4032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f8b040096a0 tx=0x7f8b04006b40).stop 2024-01-21T19:51:44.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.650+0000 7f8b0e27d700 1 -- 172.21.15.173:0/3832268737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b080fd160 msgr2=0x7f8b080fb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:44.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.650+0000 7f8b0e27d700 1 --2- 172.21.15.173:0/3832268737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b080fd160 0x7f8b080fb620 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f8af8009f80 tx=0x7f8af8009b30).stop 2024-01-21T19:51:44.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.650+0000 7f8b0e27d700 1 -- 172.21.15.173:0/3832268737 shutdown_connections 2024-01-21T19:51:44.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.650+0000 7f8b0e27d700 1 --2- 172.21.15.173:0/3832268737 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8af402ff60 0x7f8af4032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:44.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.650+0000 7f8b0e27d700 1 --2- 172.21.15.173:0/3832268737 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8b080fd160 0x7f8b080fb620 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:44.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.650+0000 7f8b0e27d700 1 -- 172.21.15.173:0/3832268737 >> 172.21.15.173:0/3832268737 conn(0x7f8b080f09a0 msgr2=0x7f8b0805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:44.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.650+0000 7f8b0e27d700 1 -- 172.21.15.173:0/3832268737 shutdown_connections 2024-01-21T19:51:44.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:44.650+0000 7f8b0e27d700 1 -- 172.21.15.173:0/3832268737 wait complete. 2024-01-21T19:51:44.643 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:45.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:44 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:42.962364+0000 mgr.smithi169.espgue (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:45.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:44 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:44.649154+0000 mon.smithi169 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.173:0/3832268737' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:51:46.054 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:51:46.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:51:47.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:46 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:44.962698+0000 mgr.smithi169.espgue (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:47.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.320+0000 7f5577fff700 1 -- 172.21.15.173:0/1616800072 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55780f51a0 msgr2=0x7f55780f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:47.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.320+0000 7f5577fff700 1 --2- 172.21.15.173:0/1616800072 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55780f51a0 0x7f55780f55c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f5568009fb0 tx=0x7f5568009b30).stop 2024-01-21T19:51:47.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.320+0000 7f5577fff700 1 -- 172.21.15.173:0/1616800072 shutdown_connections 2024-01-21T19:51:47.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.320+0000 7f5577fff700 1 --2- 172.21.15.173:0/1616800072 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55780f51a0 0x7f55780f55c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:47.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.320+0000 7f5577fff700 1 -- 172.21.15.173:0/1616800072 >> 172.21.15.173:0/1616800072 conn(0x7f55780f09a0 msgr2=0x7f55780f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:47.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.320+0000 7f5577fff700 1 -- 172.21.15.173:0/1616800072 shutdown_connections 2024-01-21T19:51:47.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.320+0000 7f5577fff700 1 -- 172.21.15.173:0/1616800072 wait complete. 2024-01-21T19:51:47.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.321+0000 7f5577fff700 1 Processor -- start 2024-01-21T19:51:47.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.321+0000 7f5577fff700 1 -- start start 2024-01-21T19:51:47.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.321+0000 7f5577fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55780f51a0 0x7f55781064d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:47.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.321+0000 7f5577fff700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55781069c0 con 0x7f55780f51a0 2024-01-21T19:51:47.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.321+0000 7f5576ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55780f51a0 0x7f55781064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:47.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.321+0000 7f5576ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55780f51a0 0x7f55781064d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:56118/0 (socket says 172.21.15.173:56118) 2024-01-21T19:51:47.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.321+0000 7f5576ffd700 1 -- 172.21.15.173:0/3377793355 learned_addr learned my addr 172.21.15.173:0/3377793355 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:51:47.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.321+0000 7f5576ffd700 1 -- 172.21.15.173:0/3377793355 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5568005040 con 0x7f55780f51a0 2024-01-21T19:51:47.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.321+0000 7f5576ffd700 1 --2- 172.21.15.173:0/3377793355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55780f51a0 0x7f55781064d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f5568013510 tx=0x7f556800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:47.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.322+0000 7f555f7fe700 1 -- 172.21.15.173:0/3377793355 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5568009320 con 0x7f55780f51a0 2024-01-21T19:51:47.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.322+0000 7f5577fff700 1 -- 172.21.15.173:0/3377793355 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5578106bc0 con 0x7f55780f51a0 2024-01-21T19:51:47.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.322+0000 7f555f7fe700 1 -- 172.21.15.173:0/3377793355 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5568013f00 con 0x7f55780f51a0 2024-01-21T19:51:47.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.322+0000 7f555f7fe700 1 -- 172.21.15.173:0/3377793355 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5568014dc0 con 0x7f55780f51a0 2024-01-21T19:51:47.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.322+0000 7f5577fff700 1 -- 172.21.15.173:0/3377793355 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5578107050 con 0x7f55780f51a0 2024-01-21T19:51:47.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.323+0000 7f555f7fe700 1 -- 172.21.15.173:0/3377793355 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f556801d3d0 con 0x7f55780f51a0 2024-01-21T19:51:47.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.323+0000 7f555f7fe700 1 --2- 172.21.15.173:0/3377793355 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f556002ff60 0x7f5560032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:47.316 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.323+0000 7f5577fff700 1 -- 172.21.15.173:0/3377793355 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5578045ad0 con 0x7f55780f51a0 2024-01-21T19:51:47.316 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.323+0000 7f555f7fe700 1 -- 172.21.15.173:0/3377793355 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5568041040 con 0x7f55780f51a0 2024-01-21T19:51:47.316 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.323+0000 7f55767fc700 1 --2- 172.21.15.173:0/3377793355 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f556002ff60 0x7f5560032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:47.316 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.323+0000 7f55767fc700 1 --2- 172.21.15.173:0/3377793355 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f556002ff60 0x7f5560032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f556c0096a0 tx=0x7f556c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:47.319 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.328+0000 7f555f7fe700 1 -- 172.21.15.173:0/3377793355 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5568012920 con 0x7f55780f51a0 2024-01-21T19:51:47.524 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.532+0000 7f5577fff700 1 -- 172.21.15.173:0/3377793355 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5578107ea0 con 0x7f55780f51a0 2024-01-21T19:51:47.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.533+0000 7f555f7fe700 1 -- 172.21.15.173:0/3377793355 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f556802b020 con 0x7f55780f51a0 2024-01-21T19:51:47.525 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:51:47.525 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:51:47.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.535+0000 7f5577fff700 1 -- 172.21.15.173:0/3377793355 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f556002ff60 msgr2=0x7f5560032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:47.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.535+0000 7f5577fff700 1 --2- 172.21.15.173:0/3377793355 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f556002ff60 0x7f5560032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f556c0096a0 tx=0x7f556c006b40).stop 2024-01-21T19:51:47.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.535+0000 7f5577fff700 1 -- 172.21.15.173:0/3377793355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55780f51a0 msgr2=0x7f55781064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:47.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.535+0000 7f5577fff700 1 --2- 172.21.15.173:0/3377793355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55780f51a0 0x7f55781064d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f5568013510 tx=0x7f556800b040).stop 2024-01-21T19:51:47.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.535+0000 7f5577fff700 1 -- 172.21.15.173:0/3377793355 shutdown_connections 2024-01-21T19:51:47.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.535+0000 7f5577fff700 1 --2- 172.21.15.173:0/3377793355 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f556002ff60 0x7f5560032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:47.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.535+0000 7f5577fff700 1 --2- 172.21.15.173:0/3377793355 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f55780f51a0 0x7f55781064d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:47.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.535+0000 7f5577fff700 1 -- 172.21.15.173:0/3377793355 >> 172.21.15.173:0/3377793355 conn(0x7f55780f09a0 msgr2=0x7f557805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:47.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.535+0000 7f5577fff700 1 -- 172.21.15.173:0/3377793355 shutdown_connections 2024-01-21T19:51:47.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:47.535+0000 7f5577fff700 1 -- 172.21.15.173:0/3377793355 wait complete. 2024-01-21T19:51:47.528 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:48.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:47 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:47.533702+0000 mon.smithi169 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.173:0/3377793355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:51:48.935 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:51:48.936 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:51:49.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:46.962927+0000 mgr.smithi169.espgue (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:50.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.229+0000 7ffb708a4700 1 -- 172.21.15.173:0/123388842 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb6c0f51a0 msgr2=0x7ffb6c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:50.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.229+0000 7ffb708a4700 1 --2- 172.21.15.173:0/123388842 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb6c0f51a0 0x7ffb6c0f55c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7ffb5c009fb0 tx=0x7ffb5c009b30).stop 2024-01-21T19:51:50.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.229+0000 7ffb708a4700 1 -- 172.21.15.173:0/123388842 shutdown_connections 2024-01-21T19:51:50.227 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.229+0000 7ffb708a4700 1 --2- 172.21.15.173:0/123388842 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb6c0f51a0 0x7ffb6c0f55c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:50.227 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.229+0000 7ffb708a4700 1 -- 172.21.15.173:0/123388842 >> 172.21.15.173:0/123388842 conn(0x7ffb6c0f09a0 msgr2=0x7ffb6c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:50.227 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.229+0000 7ffb708a4700 1 -- 172.21.15.173:0/123388842 shutdown_connections 2024-01-21T19:51:50.227 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.229+0000 7ffb708a4700 1 -- 172.21.15.173:0/123388842 wait complete. 2024-01-21T19:51:50.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.230+0000 7ffb708a4700 1 Processor -- start 2024-01-21T19:51:50.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.230+0000 7ffb708a4700 1 -- start start 2024-01-21T19:51:50.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.230+0000 7ffb708a4700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb6c0f51a0 0x7ffb6c1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:50.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.230+0000 7ffb708a4700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c1058a0 con 0x7ffb6c0f51a0 2024-01-21T19:51:50.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.231+0000 7ffb6ad9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb6c0f51a0 0x7ffb6c1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:50.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.231+0000 7ffb6ad9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb6c0f51a0 0x7ffb6c1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:56140/0 (socket says 172.21.15.173:56140) 2024-01-21T19:51:50.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.231+0000 7ffb6ad9d700 1 -- 172.21.15.173:0/2536285816 learned_addr learned my addr 172.21.15.173:0/2536285816 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:51:50.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.231+0000 7ffb6ad9d700 1 -- 172.21.15.173:0/2536285816 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb5c005040 con 0x7ffb6c0f51a0 2024-01-21T19:51:50.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.231+0000 7ffb6ad9d700 1 --2- 172.21.15.173:0/2536285816 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb6c0f51a0 0x7ffb6c1053b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7ffb5c013510 tx=0x7ffb5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:50.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.231+0000 7ffb537fe700 1 -- 172.21.15.173:0/2536285816 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb5c009320 con 0x7ffb6c0f51a0 2024-01-21T19:51:50.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.231+0000 7ffb708a4700 1 -- 172.21.15.173:0/2536285816 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb6c105aa0 con 0x7ffb6c0f51a0 2024-01-21T19:51:50.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.231+0000 7ffb537fe700 1 -- 172.21.15.173:0/2536285816 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb5c013f00 con 0x7ffb6c0f51a0 2024-01-21T19:51:50.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.231+0000 7ffb537fe700 1 -- 172.21.15.173:0/2536285816 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb5c014dc0 con 0x7ffb6c0f51a0 2024-01-21T19:51:50.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.232+0000 7ffb708a4700 1 -- 172.21.15.173:0/2536285816 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb6c105f30 con 0x7ffb6c0f51a0 2024-01-21T19:51:50.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.232+0000 7ffb537fe700 1 -- 172.21.15.173:0/2536285816 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffb5c01d3d0 con 0x7ffb6c0f51a0 2024-01-21T19:51:50.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.232+0000 7ffb708a4700 1 -- 172.21.15.173:0/2536285816 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb6c045ad0 con 0x7ffb6c0f51a0 2024-01-21T19:51:50.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.232+0000 7ffb537fe700 1 --2- 172.21.15.173:0/2536285816 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ffb5402ff60 0x7ffb54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:50.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.232+0000 7ffb537fe700 1 -- 172.21.15.173:0/2536285816 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffb5c041320 con 0x7ffb6c0f51a0 2024-01-21T19:51:50.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.233+0000 7ffb6a59c700 1 --2- 172.21.15.173:0/2536285816 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ffb5402ff60 0x7ffb54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:50.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.233+0000 7ffb6a59c700 1 --2- 172.21.15.173:0/2536285816 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ffb5402ff60 0x7ffb54032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7ffb600096a0 tx=0x7ffb60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:50.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.237+0000 7ffb537fe700 1 -- 172.21.15.173:0/2536285816 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffb5c012510 con 0x7ffb6c0f51a0 2024-01-21T19:51:50.433 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.441+0000 7ffb708a4700 1 -- 172.21.15.173:0/2536285816 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb6c106de0 con 0x7ffb6c0f51a0 2024-01-21T19:51:50.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.442+0000 7ffb537fe700 1 -- 172.21.15.173:0/2536285816 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffb5c00fbd0 con 0x7ffb6c0f51a0 2024-01-21T19:51:50.435 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:51:50.435 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:51:50.437 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.443+0000 7ffb708a4700 1 -- 172.21.15.173:0/2536285816 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ffb5402ff60 msgr2=0x7ffb54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:50.437 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.443+0000 7ffb708a4700 1 --2- 172.21.15.173:0/2536285816 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ffb5402ff60 0x7ffb54032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7ffb600096a0 tx=0x7ffb60006b40).stop 2024-01-21T19:51:50.437 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.443+0000 7ffb708a4700 1 -- 172.21.15.173:0/2536285816 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb6c0f51a0 msgr2=0x7ffb6c1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:50.438 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.443+0000 7ffb708a4700 1 --2- 172.21.15.173:0/2536285816 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb6c0f51a0 0x7ffb6c1053b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7ffb5c013510 tx=0x7ffb5c00b040).stop 2024-01-21T19:51:50.438 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.443+0000 7ffb708a4700 1 -- 172.21.15.173:0/2536285816 shutdown_connections 2024-01-21T19:51:50.438 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.443+0000 7ffb708a4700 1 --2- 172.21.15.173:0/2536285816 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ffb5402ff60 0x7ffb54032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:50.438 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.443+0000 7ffb708a4700 1 --2- 172.21.15.173:0/2536285816 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffb6c0f51a0 0x7ffb6c1053b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:50.438 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.443+0000 7ffb708a4700 1 -- 172.21.15.173:0/2536285816 >> 172.21.15.173:0/2536285816 conn(0x7ffb6c0f09a0 msgr2=0x7ffb6c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:50.438 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.443+0000 7ffb708a4700 1 -- 172.21.15.173:0/2536285816 shutdown_connections 2024-01-21T19:51:50.439 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:50.443+0000 7ffb708a4700 1 -- 172.21.15.173:0/2536285816 wait complete. 2024-01-21T19:51:50.439 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:51.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:50 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:48.963271+0000 mgr.smithi169.espgue (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:51.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:50 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:50.442500+0000 mon.smithi169 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.173:0/2536285816' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:51:51.870 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:51:51.871 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:51:53.119 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.127+0000 7fc93643b700 1 -- 172.21.15.173:0/482743240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9300f6510 msgr2=0x7fc9300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:53.120 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.127+0000 7fc93643b700 1 --2- 172.21.15.173:0/482743240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9300f6510 0x7fc9300f6930 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fc924009fb0 tx=0x7fc924009b30).stop 2024-01-21T19:51:53.120 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.127+0000 7fc93643b700 1 -- 172.21.15.173:0/482743240 shutdown_connections 2024-01-21T19:51:53.120 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.127+0000 7fc93643b700 1 --2- 172.21.15.173:0/482743240 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9300f6510 0x7fc9300f6930 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:53.120 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.127+0000 7fc93643b700 1 -- 172.21.15.173:0/482743240 >> 172.21.15.173:0/482743240 conn(0x7fc9300f1c70 msgr2=0x7fc9300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:53.120 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.127+0000 7fc93643b700 1 -- 172.21.15.173:0/482743240 shutdown_connections 2024-01-21T19:51:53.120 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.127+0000 7fc93643b700 1 -- 172.21.15.173:0/482743240 wait complete. 2024-01-21T19:51:53.120 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.127+0000 7fc93643b700 1 Processor -- start 2024-01-21T19:51:53.122 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.128+0000 7fc93643b700 1 -- start start 2024-01-21T19:51:53.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.128+0000 7fc93643b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9300f6510 0x7fc930106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:53.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.128+0000 7fc93643b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc930106c20 con 0x7fc9300f6510 2024-01-21T19:51:53.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.128+0000 7fc92ffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9300f6510 0x7fc930106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:53.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.128+0000 7fc92ffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9300f6510 0x7fc930106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:56162/0 (socket says 172.21.15.173:56162) 2024-01-21T19:51:53.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.128+0000 7fc92ffff700 1 -- 172.21.15.173:0/136567976 learned_addr learned my addr 172.21.15.173:0/136567976 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:51:53.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.128+0000 7fc92ffff700 1 -- 172.21.15.173:0/136567976 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc924005040 con 0x7fc9300f6510 2024-01-21T19:51:53.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.129+0000 7fc92ffff700 1 --2- 172.21.15.173:0/136567976 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9300f6510 0x7fc930106730 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fc924013510 tx=0x7fc92400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:53.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.129+0000 7fc92cff9700 1 -- 172.21.15.173:0/136567976 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc924009320 con 0x7fc9300f6510 2024-01-21T19:51:53.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.129+0000 7fc93643b700 1 -- 172.21.15.173:0/136567976 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc930106e20 con 0x7fc9300f6510 2024-01-21T19:51:53.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.129+0000 7fc92cff9700 1 -- 172.21.15.173:0/136567976 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc924013f00 con 0x7fc9300f6510 2024-01-21T19:51:53.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.129+0000 7fc92cff9700 1 -- 172.21.15.173:0/136567976 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc924014d90 con 0x7fc9300f6510 2024-01-21T19:51:53.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.129+0000 7fc93643b700 1 -- 172.21.15.173:0/136567976 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9301072b0 con 0x7fc9300f6510 2024-01-21T19:51:53.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.130+0000 7fc92cff9700 1 -- 172.21.15.173:0/136567976 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc9240095f0 con 0x7fc9300f6510 2024-01-21T19:51:53.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.130+0000 7fc93643b700 1 -- 172.21.15.173:0/136567976 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc930045ad0 con 0x7fc9300f6510 2024-01-21T19:51:53.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.130+0000 7fc92cff9700 1 --2- 172.21.15.173:0/136567976 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc91802ff60 0x7fc918032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:53.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.130+0000 7fc92cff9700 1 -- 172.21.15.173:0/136567976 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc924040fe0 con 0x7fc9300f6510 2024-01-21T19:51:53.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.130+0000 7fc92f7fe700 1 --2- 172.21.15.173:0/136567976 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc91802ff60 0x7fc918032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:53.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.130+0000 7fc92f7fe700 1 --2- 172.21.15.173:0/136567976 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc91802ff60 0x7fc918032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fc9200096a0 tx=0x7fc920006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:53.128 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.136+0000 7fc92cff9700 1 -- 172.21.15.173:0/136567976 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc92401d7f0 con 0x7fc9300f6510 2024-01-21T19:51:53.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:52 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:50.963655+0000 mgr.smithi169.espgue (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:53.333 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.341+0000 7fc93643b700 1 -- 172.21.15.173:0/136567976 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc9300fbf60 con 0x7fc9300f6510 2024-01-21T19:51:53.333 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.342+0000 7fc92cff9700 1 -- 172.21.15.173:0/136567976 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc924010230 con 0x7fc9300f6510 2024-01-21T19:51:53.334 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:51:53.334 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:51:53.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.343+0000 7fc93643b700 1 -- 172.21.15.173:0/136567976 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc91802ff60 msgr2=0x7fc918032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:53.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.343+0000 7fc93643b700 1 --2- 172.21.15.173:0/136567976 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc91802ff60 0x7fc918032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fc9200096a0 tx=0x7fc920006b40).stop 2024-01-21T19:51:53.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.343+0000 7fc93643b700 1 -- 172.21.15.173:0/136567976 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9300f6510 msgr2=0x7fc930106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:53.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.343+0000 7fc93643b700 1 --2- 172.21.15.173:0/136567976 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9300f6510 0x7fc930106730 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fc924013510 tx=0x7fc92400b040).stop 2024-01-21T19:51:53.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.343+0000 7fc93643b700 1 -- 172.21.15.173:0/136567976 shutdown_connections 2024-01-21T19:51:53.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.343+0000 7fc93643b700 1 --2- 172.21.15.173:0/136567976 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc91802ff60 0x7fc918032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:53.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.343+0000 7fc93643b700 1 --2- 172.21.15.173:0/136567976 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc9300f6510 0x7fc930106730 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:53.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.343+0000 7fc93643b700 1 -- 172.21.15.173:0/136567976 >> 172.21.15.173:0/136567976 conn(0x7fc9300f1c70 msgr2=0x7fc9300f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:53.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.343+0000 7fc93643b700 1 -- 172.21.15.173:0/136567976 shutdown_connections 2024-01-21T19:51:53.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:53.343+0000 7fc93643b700 1 -- 172.21.15.173:0/136567976 wait complete. 2024-01-21T19:51:53.336 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:54.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:53 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:53.342255+0000 mon.smithi169 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.173:0/136567976' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:51:54.752 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:51:54.753 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:51:55.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:54 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:52.964031+0000 mgr.smithi169.espgue (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:56.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.029+0000 7fe6c0287700 1 -- 172.21.15.173:0/2578689817 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6b80f5410 msgr2=0x7fe6b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:56.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.029+0000 7fe6c0287700 1 --2- 172.21.15.173:0/2578689817 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6b80f5410 0x7fe6b80f5830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fe6a8009fb0 tx=0x7fe6a8009b30).stop 2024-01-21T19:51:56.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.029+0000 7fe6c0287700 1 -- 172.21.15.173:0/2578689817 shutdown_connections 2024-01-21T19:51:56.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.029+0000 7fe6c0287700 1 --2- 172.21.15.173:0/2578689817 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6b80f5410 0x7fe6b80f5830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:56.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.029+0000 7fe6c0287700 1 -- 172.21.15.173:0/2578689817 >> 172.21.15.173:0/2578689817 conn(0x7fe6b80f0b70 msgr2=0x7fe6b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:56.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.029+0000 7fe6c0287700 1 -- 172.21.15.173:0/2578689817 shutdown_connections 2024-01-21T19:51:56.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.029+0000 7fe6c0287700 1 -- 172.21.15.173:0/2578689817 wait complete. 2024-01-21T19:51:56.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.030+0000 7fe6c0287700 1 Processor -- start 2024-01-21T19:51:56.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.030+0000 7fe6c0287700 1 -- start start 2024-01-21T19:51:56.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.030+0000 7fe6c0287700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6b80f5410 0x7fe6b8108970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:56.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.030+0000 7fe6c0287700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b8108e60 con 0x7fe6b80f5410 2024-01-21T19:51:56.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.030+0000 7fe6be023700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6b80f5410 0x7fe6b8108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:56.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.030+0000 7fe6be023700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6b80f5410 0x7fe6b8108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:55426/0 (socket says 172.21.15.173:55426) 2024-01-21T19:51:56.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.030+0000 7fe6be023700 1 -- 172.21.15.173:0/3754918764 learned_addr learned my addr 172.21.15.173:0/3754918764 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:51:56.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.030+0000 7fe6be023700 1 -- 172.21.15.173:0/3754918764 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a8005040 con 0x7fe6b80f5410 2024-01-21T19:51:56.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.030+0000 7fe6be023700 1 --2- 172.21.15.173:0/3754918764 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6b80f5410 0x7fe6b8108970 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fe6a8009f80 tx=0x7fe6a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:56.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.031+0000 7fe6aeffd700 1 -- 172.21.15.173:0/3754918764 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6a8009320 con 0x7fe6b80f5410 2024-01-21T19:51:56.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.031+0000 7fe6c0287700 1 -- 172.21.15.173:0/3754918764 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b8109060 con 0x7fe6b80f5410 2024-01-21T19:51:56.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.031+0000 7fe6aeffd700 1 -- 172.21.15.173:0/3754918764 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe6a8013ec0 con 0x7fe6b80f5410 2024-01-21T19:51:56.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.031+0000 7fe6aeffd700 1 -- 172.21.15.173:0/3754918764 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6a801ece0 con 0x7fe6b80f5410 2024-01-21T19:51:56.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.031+0000 7fe6c0287700 1 -- 172.21.15.173:0/3754918764 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b81094f0 con 0x7fe6b80f5410 2024-01-21T19:51:56.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.031+0000 7fe6aeffd700 1 -- 172.21.15.173:0/3754918764 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe6a801e420 con 0x7fe6b80f5410 2024-01-21T19:51:56.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.031+0000 7fe6c0287700 1 -- 172.21.15.173:0/3754918764 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b8045ad0 con 0x7fe6b80f5410 2024-01-21T19:51:56.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.032+0000 7fe6aeffd700 1 --2- 172.21.15.173:0/3754918764 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe6a402ff60 0x7fe6a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:56.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.032+0000 7fe6aeffd700 1 -- 172.21.15.173:0/3754918764 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe6a8040d70 con 0x7fe6b80f5410 2024-01-21T19:51:56.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.032+0000 7fe6bd822700 1 --2- 172.21.15.173:0/3754918764 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe6a402ff60 0x7fe6a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:56.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.032+0000 7fe6bd822700 1 --2- 172.21.15.173:0/3754918764 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe6a402ff60 0x7fe6a4032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fe6b40096a0 tx=0x7fe6b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:56.028 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.036+0000 7fe6aeffd700 1 -- 172.21.15.173:0/3754918764 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe6a801e6f0 con 0x7fe6b80f5410 2024-01-21T19:51:56.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.242+0000 7fe6c0287700 1 -- 172.21.15.173:0/3754918764 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6b810a3a0 con 0x7fe6b80f5410 2024-01-21T19:51:56.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.242+0000 7fe6aeffd700 1 -- 172.21.15.173:0/3754918764 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe6a8004490 con 0x7fe6b80f5410 2024-01-21T19:51:56.235 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:51:56.235 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:51:56.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.244+0000 7fe6c0287700 1 -- 172.21.15.173:0/3754918764 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe6a402ff60 msgr2=0x7fe6a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:56.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.244+0000 7fe6c0287700 1 --2- 172.21.15.173:0/3754918764 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe6a402ff60 0x7fe6a4032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fe6b40096a0 tx=0x7fe6b4006b40).stop 2024-01-21T19:51:56.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.244+0000 7fe6c0287700 1 -- 172.21.15.173:0/3754918764 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6b80f5410 msgr2=0x7fe6b8108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:56.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.244+0000 7fe6c0287700 1 --2- 172.21.15.173:0/3754918764 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6b80f5410 0x7fe6b8108970 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fe6a8009f80 tx=0x7fe6a800b040).stop 2024-01-21T19:51:56.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.244+0000 7fe6c0287700 1 -- 172.21.15.173:0/3754918764 shutdown_connections 2024-01-21T19:51:56.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.244+0000 7fe6c0287700 1 --2- 172.21.15.173:0/3754918764 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe6a402ff60 0x7fe6a4032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:56.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.244+0000 7fe6c0287700 1 --2- 172.21.15.173:0/3754918764 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6b80f5410 0x7fe6b8108970 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:56.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.244+0000 7fe6c0287700 1 -- 172.21.15.173:0/3754918764 >> 172.21.15.173:0/3754918764 conn(0x7fe6b80f0b70 msgr2=0x7fe6b805fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:56.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.244+0000 7fe6c0287700 1 -- 172.21.15.173:0/3754918764 shutdown_connections 2024-01-21T19:51:56.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:56.244+0000 7fe6c0287700 1 -- 172.21.15.173:0/3754918764 wait complete. 2024-01-21T19:51:56.237 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:57.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:56 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:54.964317+0000 mgr.smithi169.espgue (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:51:57.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:56 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:56.243105+0000 mon.smithi169 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.173:0/3754918764' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:51:57.610 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:51:57.610 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:51:58.938 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.945+0000 7fc2e71db700 1 -- 172.21.15.173:0/2093705964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2e00f5410 msgr2=0x7fc2e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:58.938 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.945+0000 7fc2e71db700 1 --2- 172.21.15.173:0/2093705964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2e00f5410 0x7fc2e00f5830 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fc2d4009fb0 tx=0x7fc2d4009b30).stop 2024-01-21T19:51:58.938 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.945+0000 7fc2e71db700 1 -- 172.21.15.173:0/2093705964 shutdown_connections 2024-01-21T19:51:58.938 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.945+0000 7fc2e71db700 1 --2- 172.21.15.173:0/2093705964 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2e00f5410 0x7fc2e00f5830 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:58.939 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.945+0000 7fc2e71db700 1 -- 172.21.15.173:0/2093705964 >> 172.21.15.173:0/2093705964 conn(0x7fc2e00f0b70 msgr2=0x7fc2e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:58.939 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.946+0000 7fc2e71db700 1 -- 172.21.15.173:0/2093705964 shutdown_connections 2024-01-21T19:51:58.939 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.946+0000 7fc2e71db700 1 -- 172.21.15.173:0/2093705964 wait complete. 2024-01-21T19:51:58.939 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.946+0000 7fc2e71db700 1 Processor -- start 2024-01-21T19:51:58.939 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.946+0000 7fc2e71db700 1 -- start start 2024-01-21T19:51:58.939 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.946+0000 7fc2e71db700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2e00f5410 0x7fc2e01093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:58.939 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.946+0000 7fc2e71db700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e0060280 con 0x7fc2e00f5410 2024-01-21T19:51:58.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.947+0000 7fc2e4f77700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2e00f5410 0x7fc2e01093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:58.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.947+0000 7fc2e4f77700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2e00f5410 0x7fc2e01093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:55444/0 (socket says 172.21.15.173:55444) 2024-01-21T19:51:58.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.947+0000 7fc2e4f77700 1 -- 172.21.15.173:0/1426556131 learned_addr learned my addr 172.21.15.173:0/1426556131 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:51:58.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.947+0000 7fc2e4f77700 1 -- 172.21.15.173:0/1426556131 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d4005040 con 0x7fc2e00f5410 2024-01-21T19:51:58.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.947+0000 7fc2e4f77700 1 --2- 172.21.15.173:0/1426556131 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2e00f5410 0x7fc2e01093f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fc2d4014000 tx=0x7fc2d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:58.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.947+0000 7fc2dd7fa700 1 -- 172.21.15.173:0/1426556131 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2d40095f0 con 0x7fc2e00f5410 2024-01-21T19:51:58.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.947+0000 7fc2e71db700 1 -- 172.21.15.173:0/1426556131 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e0060480 con 0x7fc2e00f5410 2024-01-21T19:51:58.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.947+0000 7fc2dd7fa700 1 -- 172.21.15.173:0/1426556131 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc2d4007e70 con 0x7fc2e00f5410 2024-01-21T19:51:58.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.947+0000 7fc2dd7fa700 1 -- 172.21.15.173:0/1426556131 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2d4014ea0 con 0x7fc2e00f5410 2024-01-21T19:51:58.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.947+0000 7fc2e71db700 1 -- 172.21.15.173:0/1426556131 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e0109b20 con 0x7fc2e00f5410 2024-01-21T19:51:58.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.948+0000 7fc2dd7fa700 1 -- 172.21.15.173:0/1426556131 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc2d4025070 con 0x7fc2e00f5410 2024-01-21T19:51:58.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.948+0000 7fc2dd7fa700 1 --2- 172.21.15.173:0/1426556131 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc2d002ff60 0x7fc2d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:51:58.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.948+0000 7fc2dd7fa700 1 -- 172.21.15.173:0/1426556131 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc2d4040a70 con 0x7fc2e00f5410 2024-01-21T19:51:58.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.948+0000 7fc2e71db700 1 -- 172.21.15.173:0/1426556131 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e0059100 con 0x7fc2e00f5410 2024-01-21T19:51:58.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.949+0000 7fc2dffff700 1 --2- 172.21.15.173:0/1426556131 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc2d002ff60 0x7fc2d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:51:58.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.949+0000 7fc2dffff700 1 --2- 172.21.15.173:0/1426556131 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc2d002ff60 0x7fc2d0032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc2cc0096a0 tx=0x7fc2cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:51:58.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:58.953+0000 7fc2dd7fa700 1 -- 172.21.15.173:0/1426556131 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2d4026350 con 0x7fc2e00f5410 2024-01-21T19:51:59.150 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:59.158+0000 7fc2e71db700 1 -- 172.21.15.173:0/1426556131 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2e0045ad0 con 0x7fc2e00f5410 2024-01-21T19:51:59.150 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:59.159+0000 7fc2dd7fa700 1 -- 172.21.15.173:0/1426556131 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc2d4025300 con 0x7fc2e00f5410 2024-01-21T19:51:59.151 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:51:59.151 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:51:59.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:59.160+0000 7fc2e71db700 1 -- 172.21.15.173:0/1426556131 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc2d002ff60 msgr2=0x7fc2d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:59.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:59.160+0000 7fc2e71db700 1 --2- 172.21.15.173:0/1426556131 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc2d002ff60 0x7fc2d0032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc2cc0096a0 tx=0x7fc2cc006b40).stop 2024-01-21T19:51:59.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:59.160+0000 7fc2e71db700 1 -- 172.21.15.173:0/1426556131 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2e00f5410 msgr2=0x7fc2e01093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:51:59.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:59.160+0000 7fc2e71db700 1 --2- 172.21.15.173:0/1426556131 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2e00f5410 0x7fc2e01093f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fc2d4014000 tx=0x7fc2d400b040).stop 2024-01-21T19:51:59.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:59.161+0000 7fc2e71db700 1 -- 172.21.15.173:0/1426556131 shutdown_connections 2024-01-21T19:51:59.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:59.161+0000 7fc2e71db700 1 --2- 172.21.15.173:0/1426556131 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc2d002ff60 0x7fc2d0032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:59.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:59.161+0000 7fc2e71db700 1 --2- 172.21.15.173:0/1426556131 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2e00f5410 0x7fc2e01093f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:51:59.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:59.161+0000 7fc2e71db700 1 -- 172.21.15.173:0/1426556131 >> 172.21.15.173:0/1426556131 conn(0x7fc2e00f0b70 msgr2=0x7fc2e00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:51:59.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:59.161+0000 7fc2e71db700 1 -- 172.21.15.173:0/1426556131 shutdown_connections 2024-01-21T19:51:59.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:51:59.161+0000 7fc2e71db700 1 -- 172.21.15.173:0/1426556131 wait complete. 2024-01-21T19:51:59.153 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:51:59.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:56.964662+0000 mgr.smithi169.espgue (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:00.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:51:59 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:51:59.159214+0000 mon.smithi169 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.173:0/1426556131' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:52:00.591 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:52:00.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:52:01.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:00 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:51:58.965043+0000 mgr.smithi169.espgue (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:01.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.811+0000 7f71a2eb5700 1 -- 172.21.15.173:0/1809609540 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f719c0f5410 msgr2=0x7f719c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:01.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.811+0000 7f71a2eb5700 1 --2- 172.21.15.173:0/1809609540 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f719c0f5410 0x7f719c0f5830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f718c009fb0 tx=0x7f718c009b30).stop 2024-01-21T19:52:01.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.811+0000 7f71a2eb5700 1 -- 172.21.15.173:0/1809609540 shutdown_connections 2024-01-21T19:52:01.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.811+0000 7f71a2eb5700 1 --2- 172.21.15.173:0/1809609540 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f719c0f5410 0x7f719c0f5830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:01.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.811+0000 7f71a2eb5700 1 -- 172.21.15.173:0/1809609540 >> 172.21.15.173:0/1809609540 conn(0x7f719c0f0b70 msgr2=0x7f719c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:01.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.811+0000 7f71a2eb5700 1 -- 172.21.15.173:0/1809609540 shutdown_connections 2024-01-21T19:52:01.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.811+0000 7f71a2eb5700 1 -- 172.21.15.173:0/1809609540 wait complete. 2024-01-21T19:52:01.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.812+0000 7f71a2eb5700 1 Processor -- start 2024-01-21T19:52:01.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.812+0000 7f71a2eb5700 1 -- start start 2024-01-21T19:52:01.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.812+0000 7f71a2eb5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f719c0f5410 0x7f719c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:01.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.812+0000 7f71a2eb5700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c060220 con 0x7f719c0f5410 2024-01-21T19:52:01.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.812+0000 7f71a0c51700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f719c0f5410 0x7f719c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:01.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.812+0000 7f71a0c51700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f719c0f5410 0x7f719c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:55450/0 (socket says 172.21.15.173:55450) 2024-01-21T19:52:01.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.812+0000 7f71a0c51700 1 -- 172.21.15.173:0/2067710320 learned_addr learned my addr 172.21.15.173:0/2067710320 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:52:01.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.812+0000 7f71a0c51700 1 -- 172.21.15.173:0/2067710320 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f718c005040 con 0x7f719c0f5410 2024-01-21T19:52:01.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.812+0000 7f71a0c51700 1 --2- 172.21.15.173:0/2067710320 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f719c0f5410 0x7f719c109450 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f718c013510 tx=0x7f718c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:01.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.813+0000 7f71997fa700 1 -- 172.21.15.173:0/2067710320 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f718c009320 con 0x7f719c0f5410 2024-01-21T19:52:01.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.813+0000 7f71a2eb5700 1 -- 172.21.15.173:0/2067710320 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f719c060420 con 0x7f719c0f5410 2024-01-21T19:52:01.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.813+0000 7f71a2eb5700 1 -- 172.21.15.173:0/2067710320 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f719c109b00 con 0x7f719c0f5410 2024-01-21T19:52:01.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.813+0000 7f71997fa700 1 -- 172.21.15.173:0/2067710320 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f718c013f00 con 0x7f719c0f5410 2024-01-21T19:52:01.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.813+0000 7f71997fa700 1 -- 172.21.15.173:0/2067710320 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f718c014d60 con 0x7f719c0f5410 2024-01-21T19:52:01.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.813+0000 7f71997fa700 1 -- 172.21.15.173:0/2067710320 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f718c0263a0 con 0x7f719c0f5410 2024-01-21T19:52:01.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.814+0000 7f71997fa700 1 --2- 172.21.15.173:0/2067710320 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f718802ff60 0x7f7188032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:01.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.814+0000 7f71997fa700 1 -- 172.21.15.173:0/2067710320 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f718c041120 con 0x7f719c0f5410 2024-01-21T19:52:01.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.814+0000 7f719bfff700 1 --2- 172.21.15.173:0/2067710320 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f718802ff60 0x7f7188032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:01.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.814+0000 7f71a2eb5700 1 -- 172.21.15.173:0/2067710320 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f719c059100 con 0x7f719c0f5410 2024-01-21T19:52:01.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.814+0000 7f719bfff700 1 --2- 172.21.15.173:0/2067710320 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f718802ff60 0x7f7188032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f71840096a0 tx=0x7f7184006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:01.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:01.819+0000 7f71997fa700 1 -- 172.21.15.173:0/2067710320 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f718c01d380 con 0x7f719c0f5410 2024-01-21T19:52:02.014 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:02.022+0000 7f71a2eb5700 1 -- 172.21.15.173:0/2067710320 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f719c045ad0 con 0x7f719c0f5410 2024-01-21T19:52:02.015 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:02.023+0000 7f71997fa700 1 -- 172.21.15.173:0/2067710320 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f718c02b030 con 0x7f719c0f5410 2024-01-21T19:52:02.015 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:52:02.015 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:52:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:02.024+0000 7f71a2eb5700 1 -- 172.21.15.173:0/2067710320 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f718802ff60 msgr2=0x7f7188032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:02.024+0000 7f71a2eb5700 1 --2- 172.21.15.173:0/2067710320 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f718802ff60 0x7f7188032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f71840096a0 tx=0x7f7184006b40).stop 2024-01-21T19:52:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:02.024+0000 7f71a2eb5700 1 -- 172.21.15.173:0/2067710320 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f719c0f5410 msgr2=0x7f719c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:02.024+0000 7f71a2eb5700 1 --2- 172.21.15.173:0/2067710320 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f719c0f5410 0x7f719c109450 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f718c013510 tx=0x7f718c00b040).stop 2024-01-21T19:52:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:02.024+0000 7f71a2eb5700 1 -- 172.21.15.173:0/2067710320 shutdown_connections 2024-01-21T19:52:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:02.024+0000 7f71a2eb5700 1 --2- 172.21.15.173:0/2067710320 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f718802ff60 0x7f7188032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:02.024+0000 7f71a2eb5700 1 --2- 172.21.15.173:0/2067710320 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f719c0f5410 0x7f719c109450 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:02.024+0000 7f71a2eb5700 1 -- 172.21.15.173:0/2067710320 >> 172.21.15.173:0/2067710320 conn(0x7f719c0f0b70 msgr2=0x7f719c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:02.024+0000 7f71a2eb5700 1 -- 172.21.15.173:0/2067710320 shutdown_connections 2024-01-21T19:52:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:02.024+0000 7f71a2eb5700 1 -- 172.21.15.173:0/2067710320 wait complete. 2024-01-21T19:52:02.018 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:52:03.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:02 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:00.965380+0000 mgr.smithi169.espgue (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:03.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:02 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:02.023273+0000 mon.smithi169 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.173:0/2067710320' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:52:03.418 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:52:03.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:52:04.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.691+0000 7f4d4bffd700 1 -- 172.21.15.173:0/2703794184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d440f5410 msgr2=0x7f4d440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:04.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.691+0000 7f4d4bffd700 1 --2- 172.21.15.173:0/2703794184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d440f5410 0x7f4d440f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f4d34009fb0 tx=0x7f4d34009b30).stop 2024-01-21T19:52:04.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.691+0000 7f4d4bffd700 1 -- 172.21.15.173:0/2703794184 shutdown_connections 2024-01-21T19:52:04.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.691+0000 7f4d4bffd700 1 --2- 172.21.15.173:0/2703794184 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d440f5410 0x7f4d440f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:04.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.691+0000 7f4d4bffd700 1 -- 172.21.15.173:0/2703794184 >> 172.21.15.173:0/2703794184 conn(0x7f4d440f0b70 msgr2=0x7f4d440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:04.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.691+0000 7f4d4bffd700 1 -- 172.21.15.173:0/2703794184 shutdown_connections 2024-01-21T19:52:04.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.691+0000 7f4d4bffd700 1 -- 172.21.15.173:0/2703794184 wait complete. 2024-01-21T19:52:04.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.692+0000 7f4d4bffd700 1 Processor -- start 2024-01-21T19:52:04.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.692+0000 7f4d4bffd700 1 -- start start 2024-01-21T19:52:04.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.692+0000 7f4d4bffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d440f5410 0x7f4d44109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:04.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.692+0000 7f4d4bffd700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d44060240 con 0x7f4d440f5410 2024-01-21T19:52:04.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.692+0000 7f4d49d99700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d440f5410 0x7f4d44109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:04.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.692+0000 7f4d49d99700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d440f5410 0x7f4d44109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:54846/0 (socket says 172.21.15.173:54846) 2024-01-21T19:52:04.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.692+0000 7f4d49d99700 1 -- 172.21.15.173:0/1085626808 learned_addr learned my addr 172.21.15.173:0/1085626808 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:52:04.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.692+0000 7f4d49d99700 1 -- 172.21.15.173:0/1085626808 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d34005040 con 0x7f4d440f5410 2024-01-21T19:52:04.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.693+0000 7f4d49d99700 1 --2- 172.21.15.173:0/1085626808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d440f5410 0x7f4d44109430 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f4d34013510 tx=0x7f4d3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:04.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.693+0000 7f4d3a7fc700 1 -- 172.21.15.173:0/1085626808 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d34009320 con 0x7f4d440f5410 2024-01-21T19:52:04.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.693+0000 7f4d3a7fc700 1 -- 172.21.15.173:0/1085626808 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d34013f00 con 0x7f4d440f5410 2024-01-21T19:52:04.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.693+0000 7f4d4bffd700 1 -- 172.21.15.173:0/1085626808 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d44060440 con 0x7f4d440f5410 2024-01-21T19:52:04.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.693+0000 7f4d3a7fc700 1 -- 172.21.15.173:0/1085626808 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d34014d90 con 0x7f4d440f5410 2024-01-21T19:52:04.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.693+0000 7f4d4bffd700 1 -- 172.21.15.173:0/1085626808 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d44109ae0 con 0x7f4d440f5410 2024-01-21T19:52:04.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.694+0000 7f4d3a7fc700 1 -- 172.21.15.173:0/1085626808 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d340095f0 con 0x7f4d440f5410 2024-01-21T19:52:04.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.694+0000 7f4d4bffd700 1 -- 172.21.15.173:0/1085626808 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d44045ad0 con 0x7f4d440f5410 2024-01-21T19:52:04.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.694+0000 7f4d3a7fc700 1 --2- 172.21.15.173:0/1085626808 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4d3002ff60 0x7f4d30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:04.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.694+0000 7f4d3a7fc700 1 -- 172.21.15.173:0/1085626808 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d34041000 con 0x7f4d440f5410 2024-01-21T19:52:04.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.694+0000 7f4d49598700 1 --2- 172.21.15.173:0/1085626808 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4d3002ff60 0x7f4d30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:04.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.694+0000 7f4d49598700 1 --2- 172.21.15.173:0/1085626808 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4d3002ff60 0x7f4d30032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f4d400096a0 tx=0x7f4d40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:04.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.699+0000 7f4d3a7fc700 1 -- 172.21.15.173:0/1085626808 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d34014420 con 0x7f4d440f5410 2024-01-21T19:52:04.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.903+0000 7f4d4bffd700 1 -- 172.21.15.173:0/1085626808 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4d440fae60 con 0x7f4d440f5410 2024-01-21T19:52:04.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.904+0000 7f4d3a7fc700 1 -- 172.21.15.173:0/1085626808 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4d34010230 con 0x7f4d440f5410 2024-01-21T19:52:04.896 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:52:04.896 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:52:04.897 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.905+0000 7f4d4bffd700 1 -- 172.21.15.173:0/1085626808 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4d3002ff60 msgr2=0x7f4d30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:04.897 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.905+0000 7f4d4bffd700 1 --2- 172.21.15.173:0/1085626808 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4d3002ff60 0x7f4d30032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f4d400096a0 tx=0x7f4d40006b40).stop 2024-01-21T19:52:04.897 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.905+0000 7f4d4bffd700 1 -- 172.21.15.173:0/1085626808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d440f5410 msgr2=0x7f4d44109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:04.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.905+0000 7f4d4bffd700 1 --2- 172.21.15.173:0/1085626808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d440f5410 0x7f4d44109430 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f4d34013510 tx=0x7f4d3400b040).stop 2024-01-21T19:52:04.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.905+0000 7f4d4bffd700 1 -- 172.21.15.173:0/1085626808 shutdown_connections 2024-01-21T19:52:04.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.905+0000 7f4d4bffd700 1 --2- 172.21.15.173:0/1085626808 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4d3002ff60 0x7f4d30032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:04.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.905+0000 7f4d4bffd700 1 --2- 172.21.15.173:0/1085626808 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d440f5410 0x7f4d44109430 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:04.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.905+0000 7f4d4bffd700 1 -- 172.21.15.173:0/1085626808 >> 172.21.15.173:0/1085626808 conn(0x7f4d440f0b70 msgr2=0x7f4d440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:04.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.905+0000 7f4d4bffd700 1 -- 172.21.15.173:0/1085626808 shutdown_connections 2024-01-21T19:52:04.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:04.905+0000 7f4d4bffd700 1 -- 172.21.15.173:0/1085626808 wait complete. 2024-01-21T19:52:04.898 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:52:05.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:04 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:02.965723+0000 mgr.smithi169.espgue (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:06.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:05 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:04.904433+0000 mon.smithi169 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.173:0/1085626808' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:52:06.336 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:52:06.336 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:52:07.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:06 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:04.966124+0000 mgr.smithi169.espgue (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:07.619 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.625+0000 7fcb5d7b6700 1 -- 172.21.15.173:0/1608533210 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb580f59d0 msgr2=0x7fcb580f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:07.619 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.625+0000 7fcb5d7b6700 1 --2- 172.21.15.173:0/1608533210 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb580f59d0 0x7fcb580f5df0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fcb4c009fb0 tx=0x7fcb4c009b30).stop 2024-01-21T19:52:07.619 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.625+0000 7fcb5d7b6700 1 -- 172.21.15.173:0/1608533210 shutdown_connections 2024-01-21T19:52:07.619 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.625+0000 7fcb5d7b6700 1 --2- 172.21.15.173:0/1608533210 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb580f59d0 0x7fcb580f5df0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:07.619 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.625+0000 7fcb5d7b6700 1 -- 172.21.15.173:0/1608533210 >> 172.21.15.173:0/1608533210 conn(0x7fcb580f02e0 msgr2=0x7fcb580f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:07.620 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.625+0000 7fcb5d7b6700 1 -- 172.21.15.173:0/1608533210 shutdown_connections 2024-01-21T19:52:07.620 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.625+0000 7fcb5d7b6700 1 -- 172.21.15.173:0/1608533210 wait complete. 2024-01-21T19:52:07.620 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.626+0000 7fcb5d7b6700 1 Processor -- start 2024-01-21T19:52:07.620 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.626+0000 7fcb5d7b6700 1 -- start start 2024-01-21T19:52:07.620 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.626+0000 7fcb5d7b6700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb580f59d0 0x7fcb580fd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:07.620 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.626+0000 7fcb5d7b6700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb580fb810 con 0x7fcb580f59d0 2024-01-21T19:52:07.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.627+0000 7fcb56ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb580f59d0 0x7fcb580fd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:07.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.627+0000 7fcb56ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb580f59d0 0x7fcb580fd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:54874/0 (socket says 172.21.15.173:54874) 2024-01-21T19:52:07.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.627+0000 7fcb56ffd700 1 -- 172.21.15.173:0/2392777374 learned_addr learned my addr 172.21.15.173:0/2392777374 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:52:07.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.627+0000 7fcb56ffd700 1 -- 172.21.15.173:0/2392777374 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4c005040 con 0x7fcb580f59d0 2024-01-21T19:52:07.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.627+0000 7fcb56ffd700 1 --2- 172.21.15.173:0/2392777374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb580f59d0 0x7fcb580fd390 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fcb4c013510 tx=0x7fcb4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:07.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.627+0000 7fcb3f7fe700 1 -- 172.21.15.173:0/2392777374 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb4c009320 con 0x7fcb580f59d0 2024-01-21T19:52:07.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.627+0000 7fcb5d7b6700 1 -- 172.21.15.173:0/2392777374 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb580fb9b0 con 0x7fcb580f59d0 2024-01-21T19:52:07.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.627+0000 7fcb3f7fe700 1 -- 172.21.15.173:0/2392777374 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb4c013f00 con 0x7fcb580f59d0 2024-01-21T19:52:07.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.627+0000 7fcb3f7fe700 1 -- 172.21.15.173:0/2392777374 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb4c014d90 con 0x7fcb580f59d0 2024-01-21T19:52:07.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.627+0000 7fcb5d7b6700 1 -- 172.21.15.173:0/2392777374 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb580fbe40 con 0x7fcb580f59d0 2024-01-21T19:52:07.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.628+0000 7fcb3f7fe700 1 -- 172.21.15.173:0/2392777374 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb4c0095f0 con 0x7fcb580f59d0 2024-01-21T19:52:07.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.628+0000 7fcb5d7b6700 1 -- 172.21.15.173:0/2392777374 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb58059100 con 0x7fcb580f59d0 2024-01-21T19:52:07.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.628+0000 7fcb3f7fe700 1 --2- 172.21.15.173:0/2392777374 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fcb4002ff60 0x7fcb40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:07.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.628+0000 7fcb3f7fe700 1 -- 172.21.15.173:0/2392777374 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb4c041000 con 0x7fcb580f59d0 2024-01-21T19:52:07.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.629+0000 7fcb567fc700 1 --2- 172.21.15.173:0/2392777374 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fcb4002ff60 0x7fcb40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:07.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.629+0000 7fcb567fc700 1 --2- 172.21.15.173:0/2392777374 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fcb4002ff60 0x7fcb40032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fcb480096a0 tx=0x7fcb48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:07.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.633+0000 7fcb3f7fe700 1 -- 172.21.15.173:0/2392777374 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb4c014420 con 0x7fcb580f59d0 2024-01-21T19:52:07.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.837+0000 7fcb5d7b6700 1 -- 172.21.15.173:0/2392777374 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb580fae50 con 0x7fcb580f59d0 2024-01-21T19:52:07.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.837+0000 7fcb3f7fe700 1 -- 172.21.15.173:0/2392777374 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcb4c01d7f0 con 0x7fcb580f59d0 2024-01-21T19:52:07.830 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:52:07.831 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:52:07.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.839+0000 7fcb5d7b6700 1 -- 172.21.15.173:0/2392777374 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fcb4002ff60 msgr2=0x7fcb40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:07.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.839+0000 7fcb5d7b6700 1 --2- 172.21.15.173:0/2392777374 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fcb4002ff60 0x7fcb40032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fcb480096a0 tx=0x7fcb48006b40).stop 2024-01-21T19:52:07.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.839+0000 7fcb5d7b6700 1 -- 172.21.15.173:0/2392777374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb580f59d0 msgr2=0x7fcb580fd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:07.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.839+0000 7fcb5d7b6700 1 --2- 172.21.15.173:0/2392777374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb580f59d0 0x7fcb580fd390 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fcb4c013510 tx=0x7fcb4c00b040).stop 2024-01-21T19:52:07.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.839+0000 7fcb5d7b6700 1 -- 172.21.15.173:0/2392777374 shutdown_connections 2024-01-21T19:52:07.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.839+0000 7fcb5d7b6700 1 --2- 172.21.15.173:0/2392777374 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fcb4002ff60 0x7fcb40032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:07.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.839+0000 7fcb5d7b6700 1 --2- 172.21.15.173:0/2392777374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb580f59d0 0x7fcb580fd390 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:07.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.839+0000 7fcb5d7b6700 1 -- 172.21.15.173:0/2392777374 >> 172.21.15.173:0/2392777374 conn(0x7fcb580f02e0 msgr2=0x7fcb580f4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:07.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.839+0000 7fcb5d7b6700 1 -- 172.21.15.173:0/2392777374 shutdown_connections 2024-01-21T19:52:07.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:07.839+0000 7fcb5d7b6700 1 -- 172.21.15.173:0/2392777374 wait complete. 2024-01-21T19:52:07.833 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:52:09.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:08 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:06.966378+0000 mgr.smithi169.espgue (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:09.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:08 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:07.839070+0000 mon.smithi169 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.173:0/2392777374' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:52:09.245 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:52:09.246 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:52:10.496 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.502+0000 7fedba29a700 1 -- 172.21.15.173:0/899608989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb40f51a0 msgr2=0x7fedb40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:10.496 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.502+0000 7fedba29a700 1 --2- 172.21.15.173:0/899608989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb40f51a0 0x7fedb40f55c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7feda4009fb0 tx=0x7feda4009b30).stop 2024-01-21T19:52:10.496 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.502+0000 7fedba29a700 1 -- 172.21.15.173:0/899608989 shutdown_connections 2024-01-21T19:52:10.496 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.502+0000 7fedba29a700 1 --2- 172.21.15.173:0/899608989 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb40f51a0 0x7fedb40f55c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:10.496 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.502+0000 7fedba29a700 1 -- 172.21.15.173:0/899608989 >> 172.21.15.173:0/899608989 conn(0x7fedb40f09a0 msgr2=0x7fedb40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:10.497 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.503+0000 7fedba29a700 1 -- 172.21.15.173:0/899608989 shutdown_connections 2024-01-21T19:52:10.497 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.503+0000 7fedba29a700 1 -- 172.21.15.173:0/899608989 wait complete. 2024-01-21T19:52:10.497 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.503+0000 7fedba29a700 1 Processor -- start 2024-01-21T19:52:10.497 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.503+0000 7fedba29a700 1 -- start start 2024-01-21T19:52:10.497 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.503+0000 7fedba29a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb40f51a0 0x7fedb41053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:10.497 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.503+0000 7fedba29a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb41058a0 con 0x7fedb40f51a0 2024-01-21T19:52:10.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.504+0000 7fedb9298700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb40f51a0 0x7fedb41053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:10.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.504+0000 7fedb9298700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb40f51a0 0x7fedb41053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:54886/0 (socket says 172.21.15.173:54886) 2024-01-21T19:52:10.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.504+0000 7fedb9298700 1 -- 172.21.15.173:0/3160414018 learned_addr learned my addr 172.21.15.173:0/3160414018 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:52:10.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.504+0000 7fedb9298700 1 -- 172.21.15.173:0/3160414018 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda4005040 con 0x7fedb40f51a0 2024-01-21T19:52:10.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.504+0000 7fedb9298700 1 --2- 172.21.15.173:0/3160414018 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb40f51a0 0x7fedb41053b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7feda4013510 tx=0x7feda400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:10.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.504+0000 7feda9ffb700 1 -- 172.21.15.173:0/3160414018 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feda4009320 con 0x7fedb40f51a0 2024-01-21T19:52:10.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.504+0000 7feda9ffb700 1 -- 172.21.15.173:0/3160414018 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feda4013f00 con 0x7fedb40f51a0 2024-01-21T19:52:10.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.504+0000 7fedba29a700 1 -- 172.21.15.173:0/3160414018 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb4105aa0 con 0x7fedb40f51a0 2024-01-21T19:52:10.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.504+0000 7feda9ffb700 1 -- 172.21.15.173:0/3160414018 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feda4014dc0 con 0x7fedb40f51a0 2024-01-21T19:52:10.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.504+0000 7fedba29a700 1 -- 172.21.15.173:0/3160414018 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb4105f30 con 0x7fedb40f51a0 2024-01-21T19:52:10.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.505+0000 7feda9ffb700 1 -- 172.21.15.173:0/3160414018 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feda40095f0 con 0x7fedb40f51a0 2024-01-21T19:52:10.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.505+0000 7fedba29a700 1 -- 172.21.15.173:0/3160414018 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedb4045ad0 con 0x7fedb40f51a0 2024-01-21T19:52:10.500 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.505+0000 7feda9ffb700 1 --2- 172.21.15.173:0/3160414018 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7feda002ff60 0x7feda0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:10.500 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.505+0000 7feda9ffb700 1 -- 172.21.15.173:0/3160414018 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feda40410c0 con 0x7fedb40f51a0 2024-01-21T19:52:10.500 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.505+0000 7fedb8a97700 1 --2- 172.21.15.173:0/3160414018 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7feda002ff60 0x7feda0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:10.500 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.506+0000 7fedb8a97700 1 --2- 172.21.15.173:0/3160414018 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7feda002ff60 0x7feda0032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fedb00096a0 tx=0x7fedb0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:10.503 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.510+0000 7feda9ffb700 1 -- 172.21.15.173:0/3160414018 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feda4014420 con 0x7fedb40f51a0 2024-01-21T19:52:10.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.714+0000 7fedba29a700 1 -- 172.21.15.173:0/3160414018 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fedb4106de0 con 0x7fedb40f51a0 2024-01-21T19:52:10.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.715+0000 7feda9ffb700 1 -- 172.21.15.173:0/3160414018 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7feda4010230 con 0x7fedb40f51a0 2024-01-21T19:52:10.709 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:52:10.709 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:52:10.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.717+0000 7fedba29a700 1 -- 172.21.15.173:0/3160414018 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7feda002ff60 msgr2=0x7feda0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:10.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.717+0000 7fedba29a700 1 --2- 172.21.15.173:0/3160414018 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7feda002ff60 0x7feda0032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fedb00096a0 tx=0x7fedb0006b40).stop 2024-01-21T19:52:10.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.717+0000 7fedba29a700 1 -- 172.21.15.173:0/3160414018 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb40f51a0 msgr2=0x7fedb41053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:10.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.717+0000 7fedba29a700 1 --2- 172.21.15.173:0/3160414018 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb40f51a0 0x7fedb41053b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7feda4013510 tx=0x7feda400b040).stop 2024-01-21T19:52:10.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.717+0000 7fedba29a700 1 -- 172.21.15.173:0/3160414018 shutdown_connections 2024-01-21T19:52:10.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.717+0000 7fedba29a700 1 --2- 172.21.15.173:0/3160414018 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7feda002ff60 0x7feda0032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:10.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.717+0000 7fedba29a700 1 --2- 172.21.15.173:0/3160414018 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fedb40f51a0 0x7fedb41053b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:10.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.717+0000 7fedba29a700 1 -- 172.21.15.173:0/3160414018 >> 172.21.15.173:0/3160414018 conn(0x7fedb40f09a0 msgr2=0x7fedb40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:10.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.717+0000 7fedba29a700 1 -- 172.21.15.173:0/3160414018 shutdown_connections 2024-01-21T19:52:10.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:10.717+0000 7fedba29a700 1 -- 172.21.15.173:0/3160414018 wait complete. 2024-01-21T19:52:10.711 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:52:11.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:10 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:08.966703+0000 mgr.smithi169.espgue (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:12.149 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:52:12.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:52:12.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:11 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:10.717022+0000 mon.smithi169 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.173:0/3160414018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:52:13.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:12 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:10.967013+0000 mgr.smithi169.espgue (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:13.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.314+0000 7f644332c700 1 -- 172.21.15.173:0/2956644828 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f643c0f51a0 msgr2=0x7f643c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:13.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.314+0000 7f644332c700 1 --2- 172.21.15.173:0/2956644828 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f643c0f51a0 0x7f643c0f55c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f6438009fb0 tx=0x7f6438009b30).stop 2024-01-21T19:52:13.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.314+0000 7f644332c700 1 -- 172.21.15.173:0/2956644828 shutdown_connections 2024-01-21T19:52:13.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.314+0000 7f644332c700 1 --2- 172.21.15.173:0/2956644828 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f643c0f51a0 0x7f643c0f55c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:13.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.314+0000 7f644332c700 1 -- 172.21.15.173:0/2956644828 >> 172.21.15.173:0/2956644828 conn(0x7f643c0f09a0 msgr2=0x7f643c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:13.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.314+0000 7f644332c700 1 -- 172.21.15.173:0/2956644828 shutdown_connections 2024-01-21T19:52:13.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.314+0000 7f644332c700 1 -- 172.21.15.173:0/2956644828 wait complete. 2024-01-21T19:52:13.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.314+0000 7f644332c700 1 Processor -- start 2024-01-21T19:52:13.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.314+0000 7f644332c700 1 -- start start 2024-01-21T19:52:13.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.314+0000 7f644332c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f643c0f51a0 0x7f643c1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:13.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.314+0000 7f644332c700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643c1058a0 con 0x7f643c0f51a0 2024-01-21T19:52:13.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.315+0000 7f644232a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f643c0f51a0 0x7f643c1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:13.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.315+0000 7f644232a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f643c0f51a0 0x7f643c1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:51948/0 (socket says 172.21.15.173:51948) 2024-01-21T19:52:13.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.315+0000 7f644232a700 1 -- 172.21.15.173:0/2837181047 learned_addr learned my addr 172.21.15.173:0/2837181047 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:52:13.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.315+0000 7f644232a700 1 -- 172.21.15.173:0/2837181047 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6438005040 con 0x7f643c0f51a0 2024-01-21T19:52:13.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.315+0000 7f644232a700 1 --2- 172.21.15.173:0/2837181047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f643c0f51a0 0x7f643c1053b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f6438013510 tx=0x7f643800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:13.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.315+0000 7f6432ffd700 1 -- 172.21.15.173:0/2837181047 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6438009320 con 0x7f643c0f51a0 2024-01-21T19:52:13.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.315+0000 7f644332c700 1 -- 172.21.15.173:0/2837181047 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f643c105aa0 con 0x7f643c0f51a0 2024-01-21T19:52:13.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.315+0000 7f6432ffd700 1 -- 172.21.15.173:0/2837181047 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6438013f00 con 0x7f643c0f51a0 2024-01-21T19:52:13.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.315+0000 7f6432ffd700 1 -- 172.21.15.173:0/2837181047 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6438014dc0 con 0x7f643c0f51a0 2024-01-21T19:52:13.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.315+0000 7f644332c700 1 -- 172.21.15.173:0/2837181047 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f643c105f30 con 0x7f643c0f51a0 2024-01-21T19:52:13.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.316+0000 7f6432ffd700 1 -- 172.21.15.173:0/2837181047 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f643801d3d0 con 0x7f643c0f51a0 2024-01-21T19:52:13.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.316+0000 7f6432ffd700 1 --2- 172.21.15.173:0/2837181047 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f642802ff60 0x7f6428032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:13.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.316+0000 7f6432ffd700 1 -- 172.21.15.173:0/2837181047 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6438041070 con 0x7f643c0f51a0 2024-01-21T19:52:13.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.316+0000 7f644332c700 1 -- 172.21.15.173:0/2837181047 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f643c045ad0 con 0x7f643c0f51a0 2024-01-21T19:52:13.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.317+0000 7f6441b29700 1 --2- 172.21.15.173:0/2837181047 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f642802ff60 0x7f6428032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:13.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.317+0000 7f6441b29700 1 --2- 172.21.15.173:0/2837181047 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f642802ff60 0x7f6428032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f642c0096a0 tx=0x7f642c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:13.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.321+0000 7f6432ffd700 1 -- 172.21.15.173:0/2837181047 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6438012920 con 0x7f643c0f51a0 2024-01-21T19:52:13.523 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.530+0000 7f644332c700 1 -- 172.21.15.173:0/2837181047 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f643c106de0 con 0x7f643c0f51a0 2024-01-21T19:52:13.523 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.530+0000 7f6432ffd700 1 -- 172.21.15.173:0/2837181047 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f643802b020 con 0x7f643c0f51a0 2024-01-21T19:52:13.523 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:52:13.523 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:52:13.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.532+0000 7f644332c700 1 -- 172.21.15.173:0/2837181047 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f642802ff60 msgr2=0x7f6428032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:13.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.532+0000 7f644332c700 1 --2- 172.21.15.173:0/2837181047 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f642802ff60 0x7f6428032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f642c0096a0 tx=0x7f642c006b40).stop 2024-01-21T19:52:13.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.532+0000 7f644332c700 1 -- 172.21.15.173:0/2837181047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f643c0f51a0 msgr2=0x7f643c1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:13.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.532+0000 7f644332c700 1 --2- 172.21.15.173:0/2837181047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f643c0f51a0 0x7f643c1053b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f6438013510 tx=0x7f643800b040).stop 2024-01-21T19:52:13.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.532+0000 7f644332c700 1 -- 172.21.15.173:0/2837181047 shutdown_connections 2024-01-21T19:52:13.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.532+0000 7f644332c700 1 --2- 172.21.15.173:0/2837181047 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f642802ff60 0x7f6428032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:13.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.532+0000 7f644332c700 1 --2- 172.21.15.173:0/2837181047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f643c0f51a0 0x7f643c1053b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:13.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.532+0000 7f644332c700 1 -- 172.21.15.173:0/2837181047 >> 172.21.15.173:0/2837181047 conn(0x7f643c0f09a0 msgr2=0x7f643c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:13.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.532+0000 7f644332c700 1 -- 172.21.15.173:0/2837181047 shutdown_connections 2024-01-21T19:52:13.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:13.532+0000 7f644332c700 1 -- 172.21.15.173:0/2837181047 wait complete. 2024-01-21T19:52:13.525 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:52:14.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:13 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:13.531938+0000 mon.smithi169 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.173:0/2837181047' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:52:14.923 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:52:14.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:52:15.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:14 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:12.967332+0000 mgr.smithi169.espgue (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:16.161 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.168+0000 7fdbf1284700 1 -- 172.21.15.173:0/2189708527 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbec0f6510 msgr2=0x7fdbec0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:16.161 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.168+0000 7fdbf1284700 1 --2- 172.21.15.173:0/2189708527 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbec0f6510 0x7fdbec0f6930 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fdbe0009fb0 tx=0x7fdbe0009b30).stop 2024-01-21T19:52:16.162 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.168+0000 7fdbf1284700 1 -- 172.21.15.173:0/2189708527 shutdown_connections 2024-01-21T19:52:16.162 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.168+0000 7fdbf1284700 1 --2- 172.21.15.173:0/2189708527 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbec0f6510 0x7fdbec0f6930 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:16.162 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.168+0000 7fdbf1284700 1 -- 172.21.15.173:0/2189708527 >> 172.21.15.173:0/2189708527 conn(0x7fdbec0f1c70 msgr2=0x7fdbec0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:16.162 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.168+0000 7fdbf1284700 1 -- 172.21.15.173:0/2189708527 shutdown_connections 2024-01-21T19:52:16.162 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.168+0000 7fdbf1284700 1 -- 172.21.15.173:0/2189708527 wait complete. 2024-01-21T19:52:16.164 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.169+0000 7fdbf1284700 1 Processor -- start 2024-01-21T19:52:16.164 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.169+0000 7fdbf1284700 1 -- start start 2024-01-21T19:52:16.164 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.169+0000 7fdbf1284700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbec0f6510 0x7fdbec106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:16.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.169+0000 7fdbf1284700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbec106c20 con 0x7fdbec0f6510 2024-01-21T19:52:16.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.169+0000 7fdbeaffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbec0f6510 0x7fdbec106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:16.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.169+0000 7fdbeaffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbec0f6510 0x7fdbec106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:51968/0 (socket says 172.21.15.173:51968) 2024-01-21T19:52:16.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.169+0000 7fdbeaffd700 1 -- 172.21.15.173:0/148946560 learned_addr learned my addr 172.21.15.173:0/148946560 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:52:16.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.170+0000 7fdbeaffd700 1 -- 172.21.15.173:0/148946560 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe0005040 con 0x7fdbec0f6510 2024-01-21T19:52:16.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.170+0000 7fdbeaffd700 1 --2- 172.21.15.173:0/148946560 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbec0f6510 0x7fdbec106730 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fdbe0013510 tx=0x7fdbe000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:16.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.170+0000 7fdbd37fe700 1 -- 172.21.15.173:0/148946560 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbe0009320 con 0x7fdbec0f6510 2024-01-21T19:52:16.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.170+0000 7fdbd37fe700 1 -- 172.21.15.173:0/148946560 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdbe0013f00 con 0x7fdbec0f6510 2024-01-21T19:52:16.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.170+0000 7fdbf1284700 1 -- 172.21.15.173:0/148946560 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbec106e20 con 0x7fdbec0f6510 2024-01-21T19:52:16.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.170+0000 7fdbd37fe700 1 -- 172.21.15.173:0/148946560 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbe0014d90 con 0x7fdbec0f6510 2024-01-21T19:52:16.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.170+0000 7fdbf1284700 1 -- 172.21.15.173:0/148946560 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbec1072b0 con 0x7fdbec0f6510 2024-01-21T19:52:16.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.171+0000 7fdbd37fe700 1 -- 172.21.15.173:0/148946560 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdbe0014420 con 0x7fdbec0f6510 2024-01-21T19:52:16.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.171+0000 7fdbf1284700 1 -- 172.21.15.173:0/148946560 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbec045ad0 con 0x7fdbec0f6510 2024-01-21T19:52:16.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.171+0000 7fdbd37fe700 1 --2- 172.21.15.173:0/148946560 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fdbd402ff60 0x7fdbd4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:16.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.171+0000 7fdbd37fe700 1 -- 172.21.15.173:0/148946560 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdbe0041000 con 0x7fdbec0f6510 2024-01-21T19:52:16.166 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.171+0000 7fdbea7fc700 1 --2- 172.21.15.173:0/148946560 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fdbd402ff60 0x7fdbd4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:16.166 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.172+0000 7fdbea7fc700 1 --2- 172.21.15.173:0/148946560 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fdbd402ff60 0x7fdbd4032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fdbdc0096a0 tx=0x7fdbdc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:16.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.176+0000 7fdbd37fe700 1 -- 172.21.15.173:0/148946560 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdbe00095f0 con 0x7fdbec0f6510 2024-01-21T19:52:16.372 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.379+0000 7fdbf1284700 1 -- 172.21.15.173:0/148946560 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdbec0fbf60 con 0x7fdbec0f6510 2024-01-21T19:52:16.373 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.380+0000 7fdbd37fe700 1 -- 172.21.15.173:0/148946560 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdbe001db70 con 0x7fdbec0f6510 2024-01-21T19:52:16.373 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:52:16.373 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:52:16.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.381+0000 7fdbf1284700 1 -- 172.21.15.173:0/148946560 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fdbd402ff60 msgr2=0x7fdbd4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:16.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.381+0000 7fdbf1284700 1 --2- 172.21.15.173:0/148946560 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fdbd402ff60 0x7fdbd4032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fdbdc0096a0 tx=0x7fdbdc006b40).stop 2024-01-21T19:52:16.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.381+0000 7fdbf1284700 1 -- 172.21.15.173:0/148946560 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbec0f6510 msgr2=0x7fdbec106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:16.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.381+0000 7fdbf1284700 1 --2- 172.21.15.173:0/148946560 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbec0f6510 0x7fdbec106730 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fdbe0013510 tx=0x7fdbe000b040).stop 2024-01-21T19:52:16.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.381+0000 7fdbf1284700 1 -- 172.21.15.173:0/148946560 shutdown_connections 2024-01-21T19:52:16.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.381+0000 7fdbf1284700 1 --2- 172.21.15.173:0/148946560 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fdbd402ff60 0x7fdbd4032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:16.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.381+0000 7fdbf1284700 1 --2- 172.21.15.173:0/148946560 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdbec0f6510 0x7fdbec106730 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:16.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.381+0000 7fdbf1284700 1 -- 172.21.15.173:0/148946560 >> 172.21.15.173:0/148946560 conn(0x7fdbec0f1c70 msgr2=0x7fdbec0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:16.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.381+0000 7fdbf1284700 1 -- 172.21.15.173:0/148946560 shutdown_connections 2024-01-21T19:52:16.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:16.381+0000 7fdbf1284700 1 -- 172.21.15.173:0/148946560 wait complete. 2024-01-21T19:52:16.375 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:52:17.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:16 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:14.967631+0000 mgr.smithi169.espgue (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:17.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:16 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:16.381492+0000 mon.smithi169 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.173:0/148946560' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:52:17.811 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:52:17.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:52:19.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.110+0000 7f340b72c700 1 -- 172.21.15.173:0/2767646474 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34040f5410 msgr2=0x7f34040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:19.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.110+0000 7f340b72c700 1 --2- 172.21.15.173:0/2767646474 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34040f5410 0x7f34040f5830 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f33f4009fb0 tx=0x7f33f4009b30).stop 2024-01-21T19:52:19.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.110+0000 7f340b72c700 1 -- 172.21.15.173:0/2767646474 shutdown_connections 2024-01-21T19:52:19.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.110+0000 7f340b72c700 1 --2- 172.21.15.173:0/2767646474 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34040f5410 0x7f34040f5830 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:19.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.110+0000 7f340b72c700 1 -- 172.21.15.173:0/2767646474 >> 172.21.15.173:0/2767646474 conn(0x7f34040f0b70 msgr2=0x7f34040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:19.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.110+0000 7f340b72c700 1 -- 172.21.15.173:0/2767646474 shutdown_connections 2024-01-21T19:52:19.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.110+0000 7f340b72c700 1 -- 172.21.15.173:0/2767646474 wait complete. 2024-01-21T19:52:19.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.111+0000 7f340b72c700 1 Processor -- start 2024-01-21T19:52:19.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.111+0000 7f340b72c700 1 -- start start 2024-01-21T19:52:19.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.111+0000 7f340b72c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34040f5410 0x7f3404106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:19.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.111+0000 7f340b72c700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3404106c00 con 0x7f34040f5410 2024-01-21T19:52:19.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.111+0000 7f34094c8700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34040f5410 0x7f3404106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:19.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.111+0000 7f34094c8700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34040f5410 0x7f3404106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:51982/0 (socket says 172.21.15.173:51982) 2024-01-21T19:52:19.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.111+0000 7f34094c8700 1 -- 172.21.15.173:0/2914436750 learned_addr learned my addr 172.21.15.173:0/2914436750 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:52:19.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.111+0000 7f34094c8700 1 -- 172.21.15.173:0/2914436750 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f4005040 con 0x7f34040f5410 2024-01-21T19:52:19.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.112+0000 7f34094c8700 1 --2- 172.21.15.173:0/2914436750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34040f5410 0x7f3404106710 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f33f4014000 tx=0x7f33f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:19.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.112+0000 7f33f9ffb700 1 -- 172.21.15.173:0/2914436750 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33f40095f0 con 0x7f34040f5410 2024-01-21T19:52:19.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.112+0000 7f340b72c700 1 -- 172.21.15.173:0/2914436750 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3404106e00 con 0x7f34040f5410 2024-01-21T19:52:19.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.112+0000 7f33f9ffb700 1 -- 172.21.15.173:0/2914436750 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33f4007e70 con 0x7f34040f5410 2024-01-21T19:52:19.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.112+0000 7f33f9ffb700 1 -- 172.21.15.173:0/2914436750 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33f4014ea0 con 0x7f34040f5410 2024-01-21T19:52:19.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.112+0000 7f340b72c700 1 -- 172.21.15.173:0/2914436750 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3404107290 con 0x7f34040f5410 2024-01-21T19:52:19.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.113+0000 7f33f9ffb700 1 -- 172.21.15.173:0/2914436750 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f33f4025070 con 0x7f34040f5410 2024-01-21T19:52:19.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.113+0000 7f340b72c700 1 -- 172.21.15.173:0/2914436750 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3404045ad0 con 0x7f34040f5410 2024-01-21T19:52:19.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.113+0000 7f33f9ffb700 1 --2- 172.21.15.173:0/2914436750 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f33f002ff60 0x7f33f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:19.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.113+0000 7f33f9ffb700 1 -- 172.21.15.173:0/2914436750 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f33f4040970 con 0x7f34040f5410 2024-01-21T19:52:19.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.113+0000 7f3408cc7700 1 --2- 172.21.15.173:0/2914436750 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f33f002ff60 0x7f33f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:19.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.113+0000 7f3408cc7700 1 --2- 172.21.15.173:0/2914436750 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f33f002ff60 0x7f33f0032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f34000096a0 tx=0x7f3400006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:19.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.118+0000 7f33f9ffb700 1 -- 172.21.15.173:0/2914436750 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f33f400f4e0 con 0x7f34040f5410 2024-01-21T19:52:19.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:18 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:16.967998+0000 mgr.smithi169.espgue (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:19.317 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.324+0000 7f340b72c700 1 -- 172.21.15.173:0/2914436750 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34040fae60 con 0x7f34040f5410 2024-01-21T19:52:19.317 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.325+0000 7f33f9ffb700 1 -- 172.21.15.173:0/2914436750 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f33f400f300 con 0x7f34040f5410 2024-01-21T19:52:19.318 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:52:19.318 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:52:19.319 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.326+0000 7f340b72c700 1 -- 172.21.15.173:0/2914436750 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f33f002ff60 msgr2=0x7f33f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:19.319 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.326+0000 7f340b72c700 1 --2- 172.21.15.173:0/2914436750 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f33f002ff60 0x7f33f0032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f34000096a0 tx=0x7f3400006b40).stop 2024-01-21T19:52:19.319 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.326+0000 7f340b72c700 1 -- 172.21.15.173:0/2914436750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34040f5410 msgr2=0x7f3404106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:19.320 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.326+0000 7f340b72c700 1 --2- 172.21.15.173:0/2914436750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34040f5410 0x7f3404106710 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f33f4014000 tx=0x7f33f400b040).stop 2024-01-21T19:52:19.320 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.326+0000 7f340b72c700 1 -- 172.21.15.173:0/2914436750 shutdown_connections 2024-01-21T19:52:19.320 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.326+0000 7f340b72c700 1 --2- 172.21.15.173:0/2914436750 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f33f002ff60 0x7f33f0032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:19.320 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.326+0000 7f340b72c700 1 --2- 172.21.15.173:0/2914436750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34040f5410 0x7f3404106710 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:19.320 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.326+0000 7f340b72c700 1 -- 172.21.15.173:0/2914436750 >> 172.21.15.173:0/2914436750 conn(0x7f34040f0b70 msgr2=0x7f340405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:19.320 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.326+0000 7f340b72c700 1 -- 172.21.15.173:0/2914436750 shutdown_connections 2024-01-21T19:52:19.320 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:19.326+0000 7f340b72c700 1 -- 172.21.15.173:0/2914436750 wait complete. 2024-01-21T19:52:19.320 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:52:20.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:19 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:19.326268+0000 mon.smithi169 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.173:0/2914436750' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:52:20.717 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:52:20.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:52:21.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:20 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:18.968373+0000 mgr.smithi169.espgue (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:21.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.937+0000 7f8c0a0dd700 1 -- 172.21.15.173:0/254699725 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c040f5410 msgr2=0x7f8c040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:21.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.937+0000 7f8c0a0dd700 1 --2- 172.21.15.173:0/254699725 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c040f5410 0x7f8c040f5830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f8bf4009fb0 tx=0x7f8bf4009b30).stop 2024-01-21T19:52:21.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.938+0000 7f8c0a0dd700 1 -- 172.21.15.173:0/254699725 shutdown_connections 2024-01-21T19:52:21.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.938+0000 7f8c0a0dd700 1 --2- 172.21.15.173:0/254699725 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c040f5410 0x7f8c040f5830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:21.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.938+0000 7f8c0a0dd700 1 -- 172.21.15.173:0/254699725 >> 172.21.15.173:0/254699725 conn(0x7f8c040f0b70 msgr2=0x7f8c040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:21.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.938+0000 7f8c0a0dd700 1 -- 172.21.15.173:0/254699725 shutdown_connections 2024-01-21T19:52:21.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.938+0000 7f8c0a0dd700 1 -- 172.21.15.173:0/254699725 wait complete. 2024-01-21T19:52:21.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.938+0000 7f8c0a0dd700 1 Processor -- start 2024-01-21T19:52:21.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.938+0000 7f8c0a0dd700 1 -- start start 2024-01-21T19:52:21.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.939+0000 7f8c0a0dd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c040f5410 0x7f8c04105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:21.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.939+0000 7f8c0a0dd700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c04105b20 con 0x7f8c040f5410 2024-01-21T19:52:21.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.939+0000 7f8c037fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c040f5410 0x7f8c04105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:21.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.939+0000 7f8c037fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c040f5410 0x7f8c04105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:51996/0 (socket says 172.21.15.173:51996) 2024-01-21T19:52:21.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.939+0000 7f8c037fe700 1 -- 172.21.15.173:0/511056928 learned_addr learned my addr 172.21.15.173:0/511056928 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:52:21.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.939+0000 7f8c037fe700 1 -- 172.21.15.173:0/511056928 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf4005040 con 0x7f8c040f5410 2024-01-21T19:52:21.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.939+0000 7f8c037fe700 1 --2- 172.21.15.173:0/511056928 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c040f5410 0x7f8c04105630 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f8bf4013510 tx=0x7f8bf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:21.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.939+0000 7f8bfbfff700 1 -- 172.21.15.173:0/511056928 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8bf4009320 con 0x7f8c040f5410 2024-01-21T19:52:21.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.939+0000 7f8c0a0dd700 1 -- 172.21.15.173:0/511056928 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c04105d20 con 0x7f8c040f5410 2024-01-21T19:52:21.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.940+0000 7f8c0a0dd700 1 -- 172.21.15.173:0/511056928 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c041061b0 con 0x7f8c040f5410 2024-01-21T19:52:21.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.940+0000 7f8bfbfff700 1 -- 172.21.15.173:0/511056928 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8bf4013f00 con 0x7f8c040f5410 2024-01-21T19:52:21.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.940+0000 7f8bfbfff700 1 -- 172.21.15.173:0/511056928 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8bf4014d90 con 0x7f8c040f5410 2024-01-21T19:52:21.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.940+0000 7f8bfbfff700 1 -- 172.21.15.173:0/511056928 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8bf40263a0 con 0x7f8c040f5410 2024-01-21T19:52:21.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.940+0000 7f8c0a0dd700 1 -- 172.21.15.173:0/511056928 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8be8004fa0 con 0x7f8c040f5410 2024-01-21T19:52:21.935 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.940+0000 7f8bfbfff700 1 --2- 172.21.15.173:0/511056928 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8bf002ffb0 0x7f8bf0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:21.935 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.941+0000 7f8bfbfff700 1 -- 172.21.15.173:0/511056928 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8bf4040fe0 con 0x7f8c040f5410 2024-01-21T19:52:21.935 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.941+0000 7f8c02ffd700 1 --2- 172.21.15.173:0/511056928 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8bf002ffb0 0x7f8bf0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:21.935 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.941+0000 7f8c02ffd700 1 --2- 172.21.15.173:0/511056928 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8bf002ffb0 0x7f8bf0032470 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f8bec0096a0 tx=0x7f8bec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:21.938 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:21.945+0000 7f8bfbfff700 1 -- 172.21.15.173:0/511056928 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8bf40126f0 con 0x7f8c040f5410 2024-01-21T19:52:22.143 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:22.149+0000 7f8c0a0dd700 1 -- 172.21.15.173:0/511056928 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8be8005670 con 0x7f8c040f5410 2024-01-21T19:52:22.143 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:22.150+0000 7f8bfbfff700 1 -- 172.21.15.173:0/511056928 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8bf4012330 con 0x7f8c040f5410 2024-01-21T19:52:22.143 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:52:22.143 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:52:22.144 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:22.151+0000 7f8c0a0dd700 1 -- 172.21.15.173:0/511056928 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8bf002ffb0 msgr2=0x7f8bf0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:22.144 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:22.151+0000 7f8c0a0dd700 1 --2- 172.21.15.173:0/511056928 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8bf002ffb0 0x7f8bf0032470 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f8bec0096a0 tx=0x7f8bec006b40).stop 2024-01-21T19:52:22.144 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:22.151+0000 7f8c0a0dd700 1 -- 172.21.15.173:0/511056928 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c040f5410 msgr2=0x7f8c04105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:22.144 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:22.151+0000 7f8c0a0dd700 1 --2- 172.21.15.173:0/511056928 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c040f5410 0x7f8c04105630 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f8bf4013510 tx=0x7f8bf400b040).stop 2024-01-21T19:52:22.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:22.151+0000 7f8c0a0dd700 1 -- 172.21.15.173:0/511056928 shutdown_connections 2024-01-21T19:52:22.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:22.151+0000 7f8c0a0dd700 1 --2- 172.21.15.173:0/511056928 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8bf002ffb0 0x7f8bf0032470 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:22.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:22.151+0000 7f8c0a0dd700 1 --2- 172.21.15.173:0/511056928 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8c040f5410 0x7f8c04105630 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:22.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:22.151+0000 7f8c0a0dd700 1 -- 172.21.15.173:0/511056928 >> 172.21.15.173:0/511056928 conn(0x7f8c040f0b70 msgr2=0x7f8c040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:22.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:22.151+0000 7f8c0a0dd700 1 -- 172.21.15.173:0/511056928 shutdown_connections 2024-01-21T19:52:22.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:22.151+0000 7f8c0a0dd700 1 -- 172.21.15.173:0/511056928 wait complete. 2024-01-21T19:52:22.145 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:52:23.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:22 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:20.968642+0000 mgr.smithi169.espgue (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:23.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:22 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:22.151571+0000 mon.smithi169 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.173:0/511056928' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:52:23.573 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:52:23.574 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:52:24.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.769+0000 7f07433c9700 1 -- 172.21.15.173:0/605964230 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f073c0f6510 msgr2=0x7f073c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:24.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.769+0000 7f07433c9700 1 --2- 172.21.15.173:0/605964230 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f073c0f6510 0x7f073c0f6930 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f0730009fb0 tx=0x7f0730009b30).stop 2024-01-21T19:52:24.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.769+0000 7f07433c9700 1 -- 172.21.15.173:0/605964230 shutdown_connections 2024-01-21T19:52:24.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.769+0000 7f07433c9700 1 --2- 172.21.15.173:0/605964230 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f073c0f6510 0x7f073c0f6930 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:24.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.769+0000 7f07433c9700 1 -- 172.21.15.173:0/605964230 >> 172.21.15.173:0/605964230 conn(0x7f073c0f1c70 msgr2=0x7f073c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:24.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.769+0000 7f07433c9700 1 -- 172.21.15.173:0/605964230 shutdown_connections 2024-01-21T19:52:24.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.769+0000 7f07433c9700 1 -- 172.21.15.173:0/605964230 wait complete. 2024-01-21T19:52:24.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.770+0000 7f07433c9700 1 Processor -- start 2024-01-21T19:52:24.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.770+0000 7f07433c9700 1 -- start start 2024-01-21T19:52:24.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.770+0000 7f07433c9700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f073c0f6510 0x7f073c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:24.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.770+0000 7f07433c9700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073c106c20 con 0x7f073c0f6510 2024-01-21T19:52:24.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.770+0000 7f0741165700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f073c0f6510 0x7f073c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:24.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.770+0000 7f0741165700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f073c0f6510 0x7f073c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:40954/0 (socket says 172.21.15.173:40954) 2024-01-21T19:52:24.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.770+0000 7f0741165700 1 -- 172.21.15.173:0/1057012712 learned_addr learned my addr 172.21.15.173:0/1057012712 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:52:24.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.770+0000 7f0741165700 1 -- 172.21.15.173:0/1057012712 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0730005040 con 0x7f073c0f6510 2024-01-21T19:52:24.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.771+0000 7f0741165700 1 --2- 172.21.15.173:0/1057012712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f073c0f6510 0x7f073c106730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f0730013510 tx=0x7f073000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:24.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.771+0000 7f072dffb700 1 -- 172.21.15.173:0/1057012712 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0730009320 con 0x7f073c0f6510 2024-01-21T19:52:24.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.771+0000 7f07433c9700 1 -- 172.21.15.173:0/1057012712 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f073c106e20 con 0x7f073c0f6510 2024-01-21T19:52:24.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.771+0000 7f072dffb700 1 -- 172.21.15.173:0/1057012712 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0730013f00 con 0x7f073c0f6510 2024-01-21T19:52:24.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.771+0000 7f072dffb700 1 -- 172.21.15.173:0/1057012712 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0730014d90 con 0x7f073c0f6510 2024-01-21T19:52:24.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.771+0000 7f07433c9700 1 -- 172.21.15.173:0/1057012712 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f073c1072b0 con 0x7f073c0f6510 2024-01-21T19:52:24.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.772+0000 7f072dffb700 1 -- 172.21.15.173:0/1057012712 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07300263a0 con 0x7f073c0f6510 2024-01-21T19:52:24.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.772+0000 7f07433c9700 1 -- 172.21.15.173:0/1057012712 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f073c045ad0 con 0x7f073c0f6510 2024-01-21T19:52:24.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.772+0000 7f072dffb700 1 --2- 172.21.15.173:0/1057012712 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f072802ff60 0x7f0728032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:24.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.772+0000 7f072dffb700 1 -- 172.21.15.173:0/1057012712 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0730041390 con 0x7f073c0f6510 2024-01-21T19:52:24.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.772+0000 7f0740964700 1 --2- 172.21.15.173:0/1057012712 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f072802ff60 0x7f0728032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:24.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.772+0000 7f0740964700 1 --2- 172.21.15.173:0/1057012712 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f072802ff60 0x7f0728032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f07380096a0 tx=0x7f0738006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:24.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.777+0000 7f072dffb700 1 -- 172.21.15.173:0/1057012712 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f073001cb10 con 0x7f073c0f6510 2024-01-21T19:52:24.973 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.980+0000 7f07433c9700 1 -- 172.21.15.173:0/1057012712 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f073c0fbf60 con 0x7f073c0f6510 2024-01-21T19:52:24.973 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.981+0000 7f072dffb700 1 -- 172.21.15.173:0/1057012712 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f073001cb10 con 0x7f073c0f6510 2024-01-21T19:52:24.974 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:52:24.974 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:52:24.975 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.982+0000 7f07433c9700 1 -- 172.21.15.173:0/1057012712 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f072802ff60 msgr2=0x7f0728032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:24.976 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.982+0000 7f07433c9700 1 --2- 172.21.15.173:0/1057012712 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f072802ff60 0x7f0728032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f07380096a0 tx=0x7f0738006b40).stop 2024-01-21T19:52:24.976 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.982+0000 7f07433c9700 1 -- 172.21.15.173:0/1057012712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f073c0f6510 msgr2=0x7f073c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:24.976 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.982+0000 7f07433c9700 1 --2- 172.21.15.173:0/1057012712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f073c0f6510 0x7f073c106730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f0730013510 tx=0x7f073000b040).stop 2024-01-21T19:52:24.976 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.982+0000 7f07433c9700 1 -- 172.21.15.173:0/1057012712 shutdown_connections 2024-01-21T19:52:24.976 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.982+0000 7f07433c9700 1 --2- 172.21.15.173:0/1057012712 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f072802ff60 0x7f0728032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:24.976 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.982+0000 7f07433c9700 1 --2- 172.21.15.173:0/1057012712 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f073c0f6510 0x7f073c106730 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:24.976 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.982+0000 7f07433c9700 1 -- 172.21.15.173:0/1057012712 >> 172.21.15.173:0/1057012712 conn(0x7f073c0f1c70 msgr2=0x7f073c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:24.976 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.982+0000 7f07433c9700 1 -- 172.21.15.173:0/1057012712 shutdown_connections 2024-01-21T19:52:24.976 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:24.982+0000 7f07433c9700 1 -- 172.21.15.173:0/1057012712 wait complete. 2024-01-21T19:52:24.976 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:52:25.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:24 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:22.968897+0000 mgr.smithi169.espgue (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:26.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:25 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:24.982255+0000 mon.smithi169 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.173:0/1057012712' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:52:26.423 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:52:26.423 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:52:27.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:26 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:24.969251+0000 mgr.smithi169.espgue (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:27.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.692+0000 7f9484337700 1 -- 172.21.15.173:0/3619198372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f947c060280 msgr2=0x7f947c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:27.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.692+0000 7f9484337700 1 --2- 172.21.15.173:0/3619198372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f947c060280 0x7f947c0f90f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f946c009fb0 tx=0x7f946c009b30).stop 2024-01-21T19:52:27.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.692+0000 7f9484337700 1 -- 172.21.15.173:0/3619198372 shutdown_connections 2024-01-21T19:52:27.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.692+0000 7f9484337700 1 --2- 172.21.15.173:0/3619198372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f947c060280 0x7f947c0f90f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:27.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.692+0000 7f9484337700 1 -- 172.21.15.173:0/3619198372 >> 172.21.15.173:0/3619198372 conn(0x7f947c0f0b70 msgr2=0x7f947c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:27.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.692+0000 7f9484337700 1 -- 172.21.15.173:0/3619198372 shutdown_connections 2024-01-21T19:52:27.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.692+0000 7f9484337700 1 -- 172.21.15.173:0/3619198372 wait complete. 2024-01-21T19:52:27.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.693+0000 7f9484337700 1 Processor -- start 2024-01-21T19:52:27.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.693+0000 7f9484337700 1 -- start start 2024-01-21T19:52:27.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.693+0000 7f9484337700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f947c060280 0x7f947c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:27.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.693+0000 7f9484337700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947c106bd0 con 0x7f947c060280 2024-01-21T19:52:27.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.693+0000 7f94820d3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f947c060280 0x7f947c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:27.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.693+0000 7f94820d3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f947c060280 0x7f947c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:40964/0 (socket says 172.21.15.173:40964) 2024-01-21T19:52:27.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.693+0000 7f94820d3700 1 -- 172.21.15.173:0/4166930966 learned_addr learned my addr 172.21.15.173:0/4166930966 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:52:27.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.693+0000 7f94820d3700 1 -- 172.21.15.173:0/4166930966 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946c005040 con 0x7f947c060280 2024-01-21T19:52:27.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.693+0000 7f94820d3700 1 --2- 172.21.15.173:0/4166930966 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f947c060280 0x7f947c1066e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f946c013510 tx=0x7f946c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:27.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.694+0000 7f9472ffd700 1 -- 172.21.15.173:0/4166930966 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f946c009320 con 0x7f947c060280 2024-01-21T19:52:27.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.694+0000 7f9484337700 1 -- 172.21.15.173:0/4166930966 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f947c106dd0 con 0x7f947c060280 2024-01-21T19:52:27.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.694+0000 7f9472ffd700 1 -- 172.21.15.173:0/4166930966 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f946c013f00 con 0x7f947c060280 2024-01-21T19:52:27.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.694+0000 7f9472ffd700 1 -- 172.21.15.173:0/4166930966 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f946c014dc0 con 0x7f947c060280 2024-01-21T19:52:27.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.694+0000 7f9484337700 1 -- 172.21.15.173:0/4166930966 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f947c107260 con 0x7f947c060280 2024-01-21T19:52:27.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.694+0000 7f9472ffd700 1 -- 172.21.15.173:0/4166930966 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f946c0095f0 con 0x7f947c060280 2024-01-21T19:52:27.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.694+0000 7f9484337700 1 -- 172.21.15.173:0/4166930966 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f947c045ad0 con 0x7f947c060280 2024-01-21T19:52:27.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.695+0000 7f9472ffd700 1 --2- 172.21.15.173:0/4166930966 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f946802ff60 0x7f9468032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:27.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.695+0000 7f9472ffd700 1 -- 172.21.15.173:0/4166930966 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f946c041080 con 0x7f947c060280 2024-01-21T19:52:27.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.695+0000 7f94818d2700 1 --2- 172.21.15.173:0/4166930966 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f946802ff60 0x7f9468032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:27.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.695+0000 7f94818d2700 1 --2- 172.21.15.173:0/4166930966 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f946802ff60 0x7f9468032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f94780096a0 tx=0x7f9478006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:27.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.699+0000 7f9472ffd700 1 -- 172.21.15.173:0/4166930966 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f946c01c790 con 0x7f947c060280 2024-01-21T19:52:27.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.900+0000 7f9484337700 1 -- 172.21.15.173:0/4166930966 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f947c108110 con 0x7f947c060280 2024-01-21T19:52:27.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.901+0000 7f9472ffd700 1 -- 172.21.15.173:0/4166930966 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f946c010230 con 0x7f947c060280 2024-01-21T19:52:27.893 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:52:27.893 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:52:27.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.903+0000 7f9484337700 1 -- 172.21.15.173:0/4166930966 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f946802ff60 msgr2=0x7f9468032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:27.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.903+0000 7f9484337700 1 --2- 172.21.15.173:0/4166930966 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f946802ff60 0x7f9468032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f94780096a0 tx=0x7f9478006b40).stop 2024-01-21T19:52:27.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.903+0000 7f9484337700 1 -- 172.21.15.173:0/4166930966 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f947c060280 msgr2=0x7f947c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:27.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.903+0000 7f9484337700 1 --2- 172.21.15.173:0/4166930966 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f947c060280 0x7f947c1066e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f946c013510 tx=0x7f946c00b040).stop 2024-01-21T19:52:27.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.903+0000 7f9484337700 1 -- 172.21.15.173:0/4166930966 shutdown_connections 2024-01-21T19:52:27.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.903+0000 7f9484337700 1 --2- 172.21.15.173:0/4166930966 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f946802ff60 0x7f9468032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:27.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.903+0000 7f9484337700 1 --2- 172.21.15.173:0/4166930966 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f947c060280 0x7f947c1066e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:27.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.903+0000 7f9484337700 1 -- 172.21.15.173:0/4166930966 >> 172.21.15.173:0/4166930966 conn(0x7f947c0f0b70 msgr2=0x7f947c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:27.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.903+0000 7f9484337700 1 -- 172.21.15.173:0/4166930966 shutdown_connections 2024-01-21T19:52:27.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:27.903+0000 7f9484337700 1 -- 172.21.15.173:0/4166930966 wait complete. 2024-01-21T19:52:27.896 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:52:29.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:28 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:26.969607+0000 mgr.smithi169.espgue (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:29.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:28 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:27.901744+0000 mon.smithi169 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.173:0/4166930966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:52:29.321 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:52:29.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:52:30.592 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.599+0000 7fd5b2193700 1 -- 172.21.15.173:0/1871490428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5ac0f5410 msgr2=0x7fd5ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:30.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.599+0000 7fd5b2193700 1 --2- 172.21.15.173:0/1871490428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5ac0f5410 0x7fd5ac0f5830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fd594009fb0 tx=0x7fd594009b30).stop 2024-01-21T19:52:30.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.599+0000 7fd5b2193700 1 -- 172.21.15.173:0/1871490428 shutdown_connections 2024-01-21T19:52:30.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.599+0000 7fd5b2193700 1 --2- 172.21.15.173:0/1871490428 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5ac0f5410 0x7fd5ac0f5830 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:30.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.599+0000 7fd5b2193700 1 -- 172.21.15.173:0/1871490428 >> 172.21.15.173:0/1871490428 conn(0x7fd5ac0f0b70 msgr2=0x7fd5ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:30.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.599+0000 7fd5b2193700 1 -- 172.21.15.173:0/1871490428 shutdown_connections 2024-01-21T19:52:30.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.599+0000 7fd5b2193700 1 -- 172.21.15.173:0/1871490428 wait complete. 2024-01-21T19:52:30.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.599+0000 7fd5b2193700 1 Processor -- start 2024-01-21T19:52:30.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.600+0000 7fd5b2193700 1 -- start start 2024-01-21T19:52:30.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.600+0000 7fd5b2193700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5ac0f5410 0x7fd5ac105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:30.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.600+0000 7fd5b2193700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ac105b20 con 0x7fd5ac0f5410 2024-01-21T19:52:30.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.600+0000 7fd5ab7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5ac0f5410 0x7fd5ac105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:30.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.600+0000 7fd5ab7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5ac0f5410 0x7fd5ac105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:40990/0 (socket says 172.21.15.173:40990) 2024-01-21T19:52:30.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.600+0000 7fd5ab7fe700 1 -- 172.21.15.173:0/380898376 learned_addr learned my addr 172.21.15.173:0/380898376 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:52:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.600+0000 7fd5ab7fe700 1 -- 172.21.15.173:0/380898376 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd594005040 con 0x7fd5ac0f5410 2024-01-21T19:52:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.600+0000 7fd5ab7fe700 1 --2- 172.21.15.173:0/380898376 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5ac0f5410 0x7fd5ac105630 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fd594013510 tx=0x7fd59400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.601+0000 7fd5a3fff700 1 -- 172.21.15.173:0/380898376 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd594009320 con 0x7fd5ac0f5410 2024-01-21T19:52:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.601+0000 7fd5b2193700 1 -- 172.21.15.173:0/380898376 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5ac105d20 con 0x7fd5ac0f5410 2024-01-21T19:52:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.601+0000 7fd5a3fff700 1 -- 172.21.15.173:0/380898376 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd594013f00 con 0x7fd5ac0f5410 2024-01-21T19:52:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.601+0000 7fd5a3fff700 1 -- 172.21.15.173:0/380898376 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd594014d90 con 0x7fd5ac0f5410 2024-01-21T19:52:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.601+0000 7fd5b2193700 1 -- 172.21.15.173:0/380898376 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5ac1061b0 con 0x7fd5ac0f5410 2024-01-21T19:52:30.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.601+0000 7fd5a3fff700 1 -- 172.21.15.173:0/380898376 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd5940095f0 con 0x7fd5ac0f5410 2024-01-21T19:52:30.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.601+0000 7fd5b2193700 1 -- 172.21.15.173:0/380898376 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5ac045ad0 con 0x7fd5ac0f5410 2024-01-21T19:52:30.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.602+0000 7fd5a3fff700 1 --2- 172.21.15.173:0/380898376 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd59802ff60 0x7fd598032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:30.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.602+0000 7fd5a3fff700 1 -- 172.21.15.173:0/380898376 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd5940410a0 con 0x7fd5ac0f5410 2024-01-21T19:52:30.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.602+0000 7fd5aaffd700 1 --2- 172.21.15.173:0/380898376 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd59802ff60 0x7fd598032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:30.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.602+0000 7fd5aaffd700 1 --2- 172.21.15.173:0/380898376 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd59802ff60 0x7fd598032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fd59c0096a0 tx=0x7fd59c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:30.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.606+0000 7fd5a3fff700 1 -- 172.21.15.173:0/380898376 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd59401d7f0 con 0x7fd5ac0f5410 2024-01-21T19:52:30.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.811+0000 7fd5b2193700 1 -- 172.21.15.173:0/380898376 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd5ac0fae60 con 0x7fd5ac0f5410 2024-01-21T19:52:30.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.811+0000 7fd5a3fff700 1 -- 172.21.15.173:0/380898376 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd594010230 con 0x7fd5ac0f5410 2024-01-21T19:52:30.803 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:52:30.803 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:52:30.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.813+0000 7fd5b2193700 1 -- 172.21.15.173:0/380898376 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd59802ff60 msgr2=0x7fd598032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:30.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.813+0000 7fd5b2193700 1 --2- 172.21.15.173:0/380898376 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd59802ff60 0x7fd598032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fd59c0096a0 tx=0x7fd59c006b40).stop 2024-01-21T19:52:30.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.813+0000 7fd5b2193700 1 -- 172.21.15.173:0/380898376 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5ac0f5410 msgr2=0x7fd5ac105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:30.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.813+0000 7fd5b2193700 1 --2- 172.21.15.173:0/380898376 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5ac0f5410 0x7fd5ac105630 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fd594013510 tx=0x7fd59400b040).stop 2024-01-21T19:52:30.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.813+0000 7fd5b2193700 1 -- 172.21.15.173:0/380898376 shutdown_connections 2024-01-21T19:52:30.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.813+0000 7fd5b2193700 1 --2- 172.21.15.173:0/380898376 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd59802ff60 0x7fd598032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:30.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.813+0000 7fd5b2193700 1 --2- 172.21.15.173:0/380898376 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5ac0f5410 0x7fd5ac105630 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:30.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.813+0000 7fd5b2193700 1 -- 172.21.15.173:0/380898376 >> 172.21.15.173:0/380898376 conn(0x7fd5ac0f0b70 msgr2=0x7fd5ac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:30.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.813+0000 7fd5b2193700 1 -- 172.21.15.173:0/380898376 shutdown_connections 2024-01-21T19:52:30.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:30.813+0000 7fd5b2193700 1 -- 172.21.15.173:0/380898376 wait complete. 2024-01-21T19:52:30.806 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:52:31.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:30 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:28.969919+0000 mgr.smithi169.espgue (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:32.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:30.812050+0000 mon.smithi169 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.173:0/380898376' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:52:32.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:31.072775+0000 mon.smithi169 (mon.0) 198 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/mirror_snapshot_schedule"}]: dispatch 2024-01-21T19:52:32.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:31.076473+0000 mon.smithi169 (mon.0) 199 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/trash_purge_schedule"}]: dispatch 2024-01-21T19:52:32.220 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:52:32.221 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:52:33.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:32 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:30.970189+0000 mgr.smithi169.espgue (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:33.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.495+0000 7f1e6b560700 1 -- 172.21.15.173:0/2065519994 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e64060220 msgr2=0x7f1e640f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:33.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.495+0000 7f1e6b560700 1 --2- 172.21.15.173:0/2065519994 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e64060220 0x7f1e640f9150 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f1e54009fb0 tx=0x7f1e54009b30).stop 2024-01-21T19:52:33.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.495+0000 7f1e6b560700 1 -- 172.21.15.173:0/2065519994 shutdown_connections 2024-01-21T19:52:33.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.495+0000 7f1e6b560700 1 --2- 172.21.15.173:0/2065519994 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e64060220 0x7f1e640f9150 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:33.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.495+0000 7f1e6b560700 1 -- 172.21.15.173:0/2065519994 >> 172.21.15.173:0/2065519994 conn(0x7f1e640f0b70 msgr2=0x7f1e640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:33.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.495+0000 7f1e6b560700 1 -- 172.21.15.173:0/2065519994 shutdown_connections 2024-01-21T19:52:33.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.495+0000 7f1e6b560700 1 -- 172.21.15.173:0/2065519994 wait complete. 2024-01-21T19:52:33.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.496+0000 7f1e6b560700 1 Processor -- start 2024-01-21T19:52:33.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.496+0000 7f1e6b560700 1 -- start start 2024-01-21T19:52:33.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.496+0000 7f1e6b560700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e64060220 0x7f1e641066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:33.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.496+0000 7f1e6b560700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e64106bd0 con 0x7f1e64060220 2024-01-21T19:52:33.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.496+0000 7f1e692fc700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e64060220 0x7f1e641066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:33.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.496+0000 7f1e692fc700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e64060220 0x7f1e641066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:46320/0 (socket says 172.21.15.173:46320) 2024-01-21T19:52:33.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.496+0000 7f1e692fc700 1 -- 172.21.15.173:0/119061777 learned_addr learned my addr 172.21.15.173:0/119061777 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:52:33.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.496+0000 7f1e692fc700 1 -- 172.21.15.173:0/119061777 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e54005040 con 0x7f1e64060220 2024-01-21T19:52:33.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.496+0000 7f1e692fc700 1 --2- 172.21.15.173:0/119061777 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e64060220 0x7f1e641066e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f1e54013510 tx=0x7f1e5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:33.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.497+0000 7f1e59ffb700 1 -- 172.21.15.173:0/119061777 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e54009320 con 0x7f1e64060220 2024-01-21T19:52:33.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.497+0000 7f1e59ffb700 1 -- 172.21.15.173:0/119061777 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e54013f00 con 0x7f1e64060220 2024-01-21T19:52:33.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.497+0000 7f1e6b560700 1 -- 172.21.15.173:0/119061777 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e64106dd0 con 0x7f1e64060220 2024-01-21T19:52:33.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.497+0000 7f1e59ffb700 1 -- 172.21.15.173:0/119061777 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e54014dc0 con 0x7f1e64060220 2024-01-21T19:52:33.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.497+0000 7f1e6b560700 1 -- 172.21.15.173:0/119061777 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e64107260 con 0x7f1e64060220 2024-01-21T19:52:33.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.498+0000 7f1e59ffb700 1 -- 172.21.15.173:0/119061777 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e540095f0 con 0x7f1e64060220 2024-01-21T19:52:33.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.498+0000 7f1e6b560700 1 -- 172.21.15.173:0/119061777 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e64045ad0 con 0x7f1e64060220 2024-01-21T19:52:33.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.498+0000 7f1e59ffb700 1 --2- 172.21.15.173:0/119061777 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1e5002ff60 0x7f1e50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:33.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.498+0000 7f1e59ffb700 1 -- 172.21.15.173:0/119061777 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e540410c0 con 0x7f1e64060220 2024-01-21T19:52:33.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.498+0000 7f1e68afb700 1 --2- 172.21.15.173:0/119061777 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1e5002ff60 0x7f1e50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:33.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.498+0000 7f1e68afb700 1 --2- 172.21.15.173:0/119061777 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1e5002ff60 0x7f1e50032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f1e600096a0 tx=0x7f1e60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:33.494 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.502+0000 7f1e59ffb700 1 -- 172.21.15.173:0/119061777 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e54014420 con 0x7f1e64060220 2024-01-21T19:52:33.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.709+0000 7f1e6b560700 1 -- 172.21.15.173:0/119061777 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e64108110 con 0x7f1e64060220 2024-01-21T19:52:33.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.709+0000 7f1e59ffb700 1 -- 172.21.15.173:0/119061777 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1e54010230 con 0x7f1e64060220 2024-01-21T19:52:33.701 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:52:33.701 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:52:33.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.711+0000 7f1e6b560700 1 -- 172.21.15.173:0/119061777 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1e5002ff60 msgr2=0x7f1e50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:33.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.711+0000 7f1e6b560700 1 --2- 172.21.15.173:0/119061777 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1e5002ff60 0x7f1e50032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f1e600096a0 tx=0x7f1e60006b40).stop 2024-01-21T19:52:33.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.711+0000 7f1e6b560700 1 -- 172.21.15.173:0/119061777 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e64060220 msgr2=0x7f1e641066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:33.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.711+0000 7f1e6b560700 1 --2- 172.21.15.173:0/119061777 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e64060220 0x7f1e641066e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f1e54013510 tx=0x7f1e5400b040).stop 2024-01-21T19:52:33.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.711+0000 7f1e6b560700 1 -- 172.21.15.173:0/119061777 shutdown_connections 2024-01-21T19:52:33.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.711+0000 7f1e6b560700 1 --2- 172.21.15.173:0/119061777 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1e5002ff60 0x7f1e50032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:33.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.711+0000 7f1e6b560700 1 --2- 172.21.15.173:0/119061777 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1e64060220 0x7f1e641066e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:33.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.711+0000 7f1e6b560700 1 -- 172.21.15.173:0/119061777 >> 172.21.15.173:0/119061777 conn(0x7f1e640f0b70 msgr2=0x7f1e640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:33.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.711+0000 7f1e6b560700 1 -- 172.21.15.173:0/119061777 shutdown_connections 2024-01-21T19:52:33.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:33.711+0000 7f1e6b560700 1 -- 172.21.15.173:0/119061777 wait complete. 2024-01-21T19:52:33.704 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:52:34.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:33 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:33.710013+0000 mon.smithi169 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.173:0/119061777' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:52:35.106 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:52:35.106 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:52:35.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:34 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:32.970547+0000 mgr.smithi169.espgue (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:36.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.345+0000 7f8baef6b700 1 -- 172.21.15.173:0/2887296977 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ba80f5410 msgr2=0x7f8ba80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:36.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.345+0000 7f8baef6b700 1 --2- 172.21.15.173:0/2887296977 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ba80f5410 0x7f8ba80f5830 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f8b9c009fb0 tx=0x7f8b9c009b30).stop 2024-01-21T19:52:36.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.345+0000 7f8baef6b700 1 -- 172.21.15.173:0/2887296977 shutdown_connections 2024-01-21T19:52:36.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.345+0000 7f8baef6b700 1 --2- 172.21.15.173:0/2887296977 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ba80f5410 0x7f8ba80f5830 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:36.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.345+0000 7f8baef6b700 1 -- 172.21.15.173:0/2887296977 >> 172.21.15.173:0/2887296977 conn(0x7f8ba80f0b70 msgr2=0x7f8ba80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:36.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.345+0000 7f8baef6b700 1 -- 172.21.15.173:0/2887296977 shutdown_connections 2024-01-21T19:52:36.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.345+0000 7f8baef6b700 1 -- 172.21.15.173:0/2887296977 wait complete. 2024-01-21T19:52:36.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.345+0000 7f8baef6b700 1 Processor -- start 2024-01-21T19:52:36.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.346+0000 7f8baef6b700 1 -- start start 2024-01-21T19:52:36.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.346+0000 7f8baef6b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ba80f5410 0x7f8ba810b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:36.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.346+0000 7f8baef6b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba8107240 con 0x7f8ba80f5410 2024-01-21T19:52:36.339 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.346+0000 7f8bacd07700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ba80f5410 0x7f8ba810b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:36.339 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.346+0000 7f8bacd07700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ba80f5410 0x7f8ba810b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:46338/0 (socket says 172.21.15.173:46338) 2024-01-21T19:52:36.339 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.346+0000 7f8bacd07700 1 -- 172.21.15.173:0/2148248009 learned_addr learned my addr 172.21.15.173:0/2148248009 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:52:36.340 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.346+0000 7f8bacd07700 1 -- 172.21.15.173:0/2148248009 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9c005040 con 0x7f8ba80f5410 2024-01-21T19:52:36.340 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.346+0000 7f8bacd07700 1 --2- 172.21.15.173:0/2148248009 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ba80f5410 0x7f8ba810b850 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f8b9c013510 tx=0x7f8b9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:36.340 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.347+0000 7f8ba57fa700 1 -- 172.21.15.173:0/2148248009 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b9c009320 con 0x7f8ba80f5410 2024-01-21T19:52:36.340 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.347+0000 7f8ba57fa700 1 -- 172.21.15.173:0/2148248009 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b9c013f00 con 0x7f8ba80f5410 2024-01-21T19:52:36.340 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.347+0000 7f8baef6b700 1 -- 172.21.15.173:0/2148248009 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba8060220 con 0x7f8ba80f5410 2024-01-21T19:52:36.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.347+0000 7f8ba57fa700 1 -- 172.21.15.173:0/2148248009 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b9c014d90 con 0x7f8ba80f5410 2024-01-21T19:52:36.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.347+0000 7f8baef6b700 1 -- 172.21.15.173:0/2148248009 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba810bd40 con 0x7f8ba80f5410 2024-01-21T19:52:36.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.347+0000 7f8ba57fa700 1 -- 172.21.15.173:0/2148248009 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8b9c0095f0 con 0x7f8ba80f5410 2024-01-21T19:52:36.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.347+0000 7f8baef6b700 1 -- 172.21.15.173:0/2148248009 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ba8045ad0 con 0x7f8ba80f5410 2024-01-21T19:52:36.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.348+0000 7f8ba57fa700 1 --2- 172.21.15.173:0/2148248009 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8b9802ff60 0x7f8b98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:36.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.348+0000 7f8ba57fa700 1 -- 172.21.15.173:0/2148248009 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8b9c041000 con 0x7f8ba80f5410 2024-01-21T19:52:36.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.348+0000 7f8ba7fff700 1 --2- 172.21.15.173:0/2148248009 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8b9802ff60 0x7f8b98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:36.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.348+0000 7f8ba7fff700 1 --2- 172.21.15.173:0/2148248009 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8b9802ff60 0x7f8b98032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f8b940096a0 tx=0x7f8b94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:36.344 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.352+0000 7f8ba57fa700 1 -- 172.21.15.173:0/2148248009 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b9c014420 con 0x7f8ba80f5410 2024-01-21T19:52:36.548 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.556+0000 7f8baef6b700 1 -- 172.21.15.173:0/2148248009 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ba80fae60 con 0x7f8ba80f5410 2024-01-21T19:52:36.549 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.557+0000 7f8ba57fa700 1 -- 172.21.15.173:0/2148248009 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8b9c01d7f0 con 0x7f8ba80f5410 2024-01-21T19:52:36.549 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:52:36.549 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:52:36.550 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.558+0000 7f8baef6b700 1 -- 172.21.15.173:0/2148248009 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8b9802ff60 msgr2=0x7f8b98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:36.550 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.558+0000 7f8baef6b700 1 --2- 172.21.15.173:0/2148248009 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8b9802ff60 0x7f8b98032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f8b940096a0 tx=0x7f8b94006b40).stop 2024-01-21T19:52:36.550 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.558+0000 7f8baef6b700 1 -- 172.21.15.173:0/2148248009 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ba80f5410 msgr2=0x7f8ba810b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:36.550 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.558+0000 7f8baef6b700 1 --2- 172.21.15.173:0/2148248009 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ba80f5410 0x7f8ba810b850 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f8b9c013510 tx=0x7f8b9c00b040).stop 2024-01-21T19:52:36.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.558+0000 7f8baef6b700 1 -- 172.21.15.173:0/2148248009 shutdown_connections 2024-01-21T19:52:36.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.558+0000 7f8baef6b700 1 --2- 172.21.15.173:0/2148248009 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8b9802ff60 0x7f8b98032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:36.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.558+0000 7f8baef6b700 1 --2- 172.21.15.173:0/2148248009 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ba80f5410 0x7f8ba810b850 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:36.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.558+0000 7f8baef6b700 1 -- 172.21.15.173:0/2148248009 >> 172.21.15.173:0/2148248009 conn(0x7f8ba80f0b70 msgr2=0x7f8ba80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:36.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.558+0000 7f8baef6b700 1 -- 172.21.15.173:0/2148248009 shutdown_connections 2024-01-21T19:52:36.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:36.558+0000 7f8baef6b700 1 -- 172.21.15.173:0/2148248009 wait complete. 2024-01-21T19:52:36.551 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:52:37.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:36 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:34.970915+0000 mgr.smithi169.espgue (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:37.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:36 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:36.557615+0000 mon.smithi169 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.173:0/2148248009' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:52:37.939 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:52:37.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:52:39.132 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.139+0000 7f8921ded700 1 -- 172.21.15.173:0/2996672380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f891c0601e0 msgr2=0x7f891c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:39.132 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.139+0000 7f8921ded700 1 --2- 172.21.15.173:0/2996672380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f891c0601e0 0x7f891c0f8f20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f8910009fb0 tx=0x7f8910009b30).stop 2024-01-21T19:52:39.132 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.139+0000 7f8921ded700 1 -- 172.21.15.173:0/2996672380 shutdown_connections 2024-01-21T19:52:39.132 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.139+0000 7f8921ded700 1 --2- 172.21.15.173:0/2996672380 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f891c0601e0 0x7f891c0f8f20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:39.132 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.139+0000 7f8921ded700 1 -- 172.21.15.173:0/2996672380 >> 172.21.15.173:0/2996672380 conn(0x7f891c0f09a0 msgr2=0x7f891c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:39.132 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.140+0000 7f8921ded700 1 -- 172.21.15.173:0/2996672380 shutdown_connections 2024-01-21T19:52:39.132 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.140+0000 7f8921ded700 1 -- 172.21.15.173:0/2996672380 wait complete. 2024-01-21T19:52:39.132 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.140+0000 7f8921ded700 1 Processor -- start 2024-01-21T19:52:39.132 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.140+0000 7f8921ded700 1 -- start start 2024-01-21T19:52:39.134 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.140+0000 7f8921ded700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f891c0601e0 0x7f891c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:39.134 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.140+0000 7f8921ded700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891c1069f0 con 0x7f891c0601e0 2024-01-21T19:52:39.134 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.141+0000 7f8920deb700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f891c0601e0 0x7f891c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:39.135 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.141+0000 7f8920deb700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f891c0601e0 0x7f891c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:46360/0 (socket says 172.21.15.173:46360) 2024-01-21T19:52:39.135 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.141+0000 7f8920deb700 1 -- 172.21.15.173:0/1961983572 learned_addr learned my addr 172.21.15.173:0/1961983572 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:52:39.135 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.141+0000 7f8920deb700 1 -- 172.21.15.173:0/1961983572 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8910005040 con 0x7f891c0601e0 2024-01-21T19:52:39.135 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.141+0000 7f8920deb700 1 --2- 172.21.15.173:0/1961983572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f891c0601e0 0x7f891c106500 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f8910013510 tx=0x7f891000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:39.135 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.141+0000 7f89197fa700 1 -- 172.21.15.173:0/1961983572 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8910009320 con 0x7f891c0601e0 2024-01-21T19:52:39.135 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.141+0000 7f89197fa700 1 -- 172.21.15.173:0/1961983572 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8910013f00 con 0x7f891c0601e0 2024-01-21T19:52:39.135 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.141+0000 7f8921ded700 1 -- 172.21.15.173:0/1961983572 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f891c106bf0 con 0x7f891c0601e0 2024-01-21T19:52:39.135 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.141+0000 7f89197fa700 1 -- 172.21.15.173:0/1961983572 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8910014d90 con 0x7f891c0601e0 2024-01-21T19:52:39.135 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.141+0000 7f8921ded700 1 -- 172.21.15.173:0/1961983572 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f891c107080 con 0x7f891c0601e0 2024-01-21T19:52:39.135 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.142+0000 7f89197fa700 1 -- 172.21.15.173:0/1961983572 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8910014420 con 0x7f891c0601e0 2024-01-21T19:52:39.135 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.142+0000 7f8921ded700 1 -- 172.21.15.173:0/1961983572 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f891c045ad0 con 0x7f891c0601e0 2024-01-21T19:52:39.135 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.142+0000 7f89197fa700 1 --2- 172.21.15.173:0/1961983572 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f890c02ffb0 0x7f890c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:39.135 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.142+0000 7f89197fa700 1 -- 172.21.15.173:0/1961983572 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8910040ff0 con 0x7f891c0601e0 2024-01-21T19:52:39.135 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.142+0000 7f891bfff700 1 --2- 172.21.15.173:0/1961983572 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f890c02ffb0 0x7f890c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:39.136 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.143+0000 7f891bfff700 1 --2- 172.21.15.173:0/1961983572 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f890c02ffb0 0x7f890c032470 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f89080096a0 tx=0x7f8908006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:39.138 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.147+0000 7f89197fa700 1 -- 172.21.15.173:0/1961983572 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f891006d2b0 con 0x7f891c0601e0 2024-01-21T19:52:39.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:36.971288+0000 mgr.smithi169.espgue (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:39.343 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.351+0000 7f8921ded700 1 -- 172.21.15.173:0/1961983572 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f891c107f30 con 0x7f891c0601e0 2024-01-21T19:52:39.344 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.352+0000 7f89197fa700 1 -- 172.21.15.173:0/1961983572 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f89100095f0 con 0x7f891c0601e0 2024-01-21T19:52:39.344 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:52:39.344 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:52:39.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.354+0000 7f8921ded700 1 -- 172.21.15.173:0/1961983572 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f890c02ffb0 msgr2=0x7f890c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:39.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.354+0000 7f8921ded700 1 --2- 172.21.15.173:0/1961983572 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f890c02ffb0 0x7f890c032470 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f89080096a0 tx=0x7f8908006b40).stop 2024-01-21T19:52:39.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.354+0000 7f8921ded700 1 -- 172.21.15.173:0/1961983572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f891c0601e0 msgr2=0x7f891c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:39.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.354+0000 7f8921ded700 1 --2- 172.21.15.173:0/1961983572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f891c0601e0 0x7f891c106500 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f8910013510 tx=0x7f891000b040).stop 2024-01-21T19:52:39.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.354+0000 7f8921ded700 1 -- 172.21.15.173:0/1961983572 shutdown_connections 2024-01-21T19:52:39.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.354+0000 7f8921ded700 1 --2- 172.21.15.173:0/1961983572 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f890c02ffb0 0x7f890c032470 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:39.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.354+0000 7f8921ded700 1 --2- 172.21.15.173:0/1961983572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f891c0601e0 0x7f891c106500 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:39.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.354+0000 7f8921ded700 1 -- 172.21.15.173:0/1961983572 >> 172.21.15.173:0/1961983572 conn(0x7f891c0f09a0 msgr2=0x7f891c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:39.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.354+0000 7f8921ded700 1 -- 172.21.15.173:0/1961983572 shutdown_connections 2024-01-21T19:52:39.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:39.354+0000 7f8921ded700 1 -- 172.21.15.173:0/1961983572 wait complete. 2024-01-21T19:52:39.347 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:52:40.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:39 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:39.352825+0000 mon.smithi169 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.173:0/1961983572' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:52:40.760 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:52:40.760 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:52:41.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:40 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:38.971649+0000 mgr.smithi169.espgue (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:41.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.989+0000 7efde8a2e700 1 -- 172.21.15.173:0/2167071685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efde40f6510 msgr2=0x7efde40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:41.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.989+0000 7efde8a2e700 1 --2- 172.21.15.173:0/2167071685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efde40f6510 0x7efde40f6930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7efdcc009fb0 tx=0x7efdcc009b30).stop 2024-01-21T19:52:41.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.989+0000 7efde8a2e700 1 -- 172.21.15.173:0/2167071685 shutdown_connections 2024-01-21T19:52:41.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.989+0000 7efde8a2e700 1 --2- 172.21.15.173:0/2167071685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efde40f6510 0x7efde40f6930 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:41.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.989+0000 7efde8a2e700 1 -- 172.21.15.173:0/2167071685 >> 172.21.15.173:0/2167071685 conn(0x7efde40f1c70 msgr2=0x7efde40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:41.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.990+0000 7efde8a2e700 1 -- 172.21.15.173:0/2167071685 shutdown_connections 2024-01-21T19:52:41.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.990+0000 7efde8a2e700 1 -- 172.21.15.173:0/2167071685 wait complete. 2024-01-21T19:52:41.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.990+0000 7efde8a2e700 1 Processor -- start 2024-01-21T19:52:41.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.990+0000 7efde8a2e700 1 -- start start 2024-01-21T19:52:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.990+0000 7efde8a2e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efde40f6510 0x7efde4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.990+0000 7efde8a2e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde4106c20 con 0x7efde40f6510 2024-01-21T19:52:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.990+0000 7efde259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efde40f6510 0x7efde4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.990+0000 7efde259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efde40f6510 0x7efde4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:46376/0 (socket says 172.21.15.173:46376) 2024-01-21T19:52:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.990+0000 7efde259c700 1 -- 172.21.15.173:0/923587023 learned_addr learned my addr 172.21.15.173:0/923587023 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:52:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.991+0000 7efde259c700 1 -- 172.21.15.173:0/923587023 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdcc005040 con 0x7efde40f6510 2024-01-21T19:52:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.991+0000 7efde259c700 1 --2- 172.21.15.173:0/923587023 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efde40f6510 0x7efde4106730 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7efdcc013510 tx=0x7efdcc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.991+0000 7efddaffd700 1 -- 172.21.15.173:0/923587023 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdcc009320 con 0x7efde40f6510 2024-01-21T19:52:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.991+0000 7efddaffd700 1 -- 172.21.15.173:0/923587023 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efdcc013f00 con 0x7efde40f6510 2024-01-21T19:52:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.991+0000 7efddaffd700 1 -- 172.21.15.173:0/923587023 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdcc014d90 con 0x7efde40f6510 2024-01-21T19:52:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.991+0000 7efde8a2e700 1 -- 172.21.15.173:0/923587023 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde4106e20 con 0x7efde40f6510 2024-01-21T19:52:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.991+0000 7efde8a2e700 1 -- 172.21.15.173:0/923587023 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde41072b0 con 0x7efde40f6510 2024-01-21T19:52:41.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.992+0000 7efddaffd700 1 -- 172.21.15.173:0/923587023 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efdcc0095f0 con 0x7efde40f6510 2024-01-21T19:52:41.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.992+0000 7efde8a2e700 1 -- 172.21.15.173:0/923587023 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde4045ad0 con 0x7efde40f6510 2024-01-21T19:52:41.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.992+0000 7efddaffd700 1 --2- 172.21.15.173:0/923587023 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7efdd002ff60 0x7efdd0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:41.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.992+0000 7efddaffd700 1 -- 172.21.15.173:0/923587023 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efdcc041020 con 0x7efde40f6510 2024-01-21T19:52:41.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.992+0000 7efde1d9b700 1 --2- 172.21.15.173:0/923587023 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7efdd002ff60 0x7efdd0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:41.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.993+0000 7efde1d9b700 1 --2- 172.21.15.173:0/923587023 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7efdd002ff60 0x7efdd0032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7efdd40096a0 tx=0x7efdd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:41.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:41.997+0000 7efddaffd700 1 -- 172.21.15.173:0/923587023 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efdcc014420 con 0x7efde40f6510 2024-01-21T19:52:42.193 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:42.201+0000 7efde8a2e700 1 -- 172.21.15.173:0/923587023 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efde40fbf60 con 0x7efde40f6510 2024-01-21T19:52:42.193 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:42.202+0000 7efddaffd700 1 -- 172.21.15.173:0/923587023 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efdcc01d7f0 con 0x7efde40f6510 2024-01-21T19:52:42.194 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:52:42.194 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:52:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:42.204+0000 7efde8a2e700 1 -- 172.21.15.173:0/923587023 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7efdd002ff60 msgr2=0x7efdd0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:42.204+0000 7efde8a2e700 1 --2- 172.21.15.173:0/923587023 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7efdd002ff60 0x7efdd0032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7efdd40096a0 tx=0x7efdd4006b40).stop 2024-01-21T19:52:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:42.204+0000 7efde8a2e700 1 -- 172.21.15.173:0/923587023 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efde40f6510 msgr2=0x7efde4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:42.204+0000 7efde8a2e700 1 --2- 172.21.15.173:0/923587023 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efde40f6510 0x7efde4106730 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7efdcc013510 tx=0x7efdcc00b040).stop 2024-01-21T19:52:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:42.204+0000 7efde8a2e700 1 -- 172.21.15.173:0/923587023 shutdown_connections 2024-01-21T19:52:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:42.204+0000 7efde8a2e700 1 --2- 172.21.15.173:0/923587023 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7efdd002ff60 0x7efdd0032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:42.204+0000 7efde8a2e700 1 --2- 172.21.15.173:0/923587023 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7efde40f6510 0x7efde4106730 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:42.204+0000 7efde8a2e700 1 -- 172.21.15.173:0/923587023 >> 172.21.15.173:0/923587023 conn(0x7efde40f1c70 msgr2=0x7efde40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:42.204+0000 7efde8a2e700 1 -- 172.21.15.173:0/923587023 shutdown_connections 2024-01-21T19:52:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:42.204+0000 7efde8a2e700 1 -- 172.21.15.173:0/923587023 wait complete. 2024-01-21T19:52:42.197 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:52:43.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:40.971925+0000 mgr.smithi169.espgue (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:43.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:42.202560+0000 mon.smithi169 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.173:0/923587023' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:52:43.600 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:52:43.601 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:52:44.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.862+0000 7ff855247700 1 -- 172.21.15.173:0/49854169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8500f5410 msgr2=0x7ff8500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:44.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.862+0000 7ff855247700 1 --2- 172.21.15.173:0/49854169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8500f5410 0x7ff8500f5830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7ff840009fb0 tx=0x7ff840009b30).stop 2024-01-21T19:52:44.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.862+0000 7ff855247700 1 -- 172.21.15.173:0/49854169 shutdown_connections 2024-01-21T19:52:44.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.862+0000 7ff855247700 1 --2- 172.21.15.173:0/49854169 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8500f5410 0x7ff8500f5830 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:44.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.862+0000 7ff855247700 1 -- 172.21.15.173:0/49854169 >> 172.21.15.173:0/49854169 conn(0x7ff8500f0b70 msgr2=0x7ff8500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:44.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.862+0000 7ff855247700 1 -- 172.21.15.173:0/49854169 shutdown_connections 2024-01-21T19:52:44.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.862+0000 7ff855247700 1 -- 172.21.15.173:0/49854169 wait complete. 2024-01-21T19:52:44.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.863+0000 7ff855247700 1 Processor -- start 2024-01-21T19:52:44.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.863+0000 7ff855247700 1 -- start start 2024-01-21T19:52:44.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.863+0000 7ff855247700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8500f5410 0x7ff850109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:44.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.863+0000 7ff855247700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff850060220 con 0x7ff8500f5410 2024-01-21T19:52:44.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.863+0000 7ff84ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8500f5410 0x7ff850109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:44.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.863+0000 7ff84ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8500f5410 0x7ff850109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:54768/0 (socket says 172.21.15.173:54768) 2024-01-21T19:52:44.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.863+0000 7ff84ed9d700 1 -- 172.21.15.173:0/4120591622 learned_addr learned my addr 172.21.15.173:0/4120591622 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:52:44.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.863+0000 7ff84ed9d700 1 -- 172.21.15.173:0/4120591622 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff840005040 con 0x7ff8500f5410 2024-01-21T19:52:44.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.863+0000 7ff84ed9d700 1 --2- 172.21.15.173:0/4120591622 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8500f5410 0x7ff850109450 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7ff840013510 tx=0x7ff84000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:44.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.864+0000 7ff8377fe700 1 -- 172.21.15.173:0/4120591622 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff840009320 con 0x7ff8500f5410 2024-01-21T19:52:44.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.864+0000 7ff8377fe700 1 -- 172.21.15.173:0/4120591622 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff840013f00 con 0x7ff8500f5410 2024-01-21T19:52:44.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.864+0000 7ff8377fe700 1 -- 172.21.15.173:0/4120591622 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff840014d90 con 0x7ff8500f5410 2024-01-21T19:52:44.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.864+0000 7ff855247700 1 -- 172.21.15.173:0/4120591622 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff850060420 con 0x7ff8500f5410 2024-01-21T19:52:44.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.864+0000 7ff855247700 1 -- 172.21.15.173:0/4120591622 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff850109b00 con 0x7ff8500f5410 2024-01-21T19:52:44.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.864+0000 7ff8377fe700 1 -- 172.21.15.173:0/4120591622 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff8400095f0 con 0x7ff8500f5410 2024-01-21T19:52:44.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.864+0000 7ff855247700 1 -- 172.21.15.173:0/4120591622 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff850059100 con 0x7ff8500f5410 2024-01-21T19:52:44.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.865+0000 7ff8377fe700 1 --2- 172.21.15.173:0/4120591622 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff83802ff60 0x7ff838032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:44.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.865+0000 7ff8377fe700 1 -- 172.21.15.173:0/4120591622 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff840041000 con 0x7ff8500f5410 2024-01-21T19:52:44.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.865+0000 7ff84e59c700 1 --2- 172.21.15.173:0/4120591622 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff83802ff60 0x7ff838032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:44.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.865+0000 7ff84e59c700 1 --2- 172.21.15.173:0/4120591622 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff83802ff60 0x7ff838032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7ff8440096a0 tx=0x7ff844006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:44.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:44.870+0000 7ff8377fe700 1 -- 172.21.15.173:0/4120591622 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff84001d7f0 con 0x7ff8500f5410 2024-01-21T19:52:45.063 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:45.072+0000 7ff855247700 1 -- 172.21.15.173:0/4120591622 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff850045ad0 con 0x7ff8500f5410 2024-01-21T19:52:45.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:45.072+0000 7ff8377fe700 1 -- 172.21.15.173:0/4120591622 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff840010230 con 0x7ff8500f5410 2024-01-21T19:52:45.064 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:52:45.064 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:52:45.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:45.074+0000 7ff855247700 1 -- 172.21.15.173:0/4120591622 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff83802ff60 msgr2=0x7ff838032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:45.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:45.074+0000 7ff855247700 1 --2- 172.21.15.173:0/4120591622 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff83802ff60 0x7ff838032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7ff8440096a0 tx=0x7ff844006b40).stop 2024-01-21T19:52:45.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:45.074+0000 7ff855247700 1 -- 172.21.15.173:0/4120591622 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8500f5410 msgr2=0x7ff850109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:45.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:45.074+0000 7ff855247700 1 --2- 172.21.15.173:0/4120591622 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8500f5410 0x7ff850109450 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7ff840013510 tx=0x7ff84000b040).stop 2024-01-21T19:52:45.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:45.074+0000 7ff855247700 1 -- 172.21.15.173:0/4120591622 shutdown_connections 2024-01-21T19:52:45.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:45.074+0000 7ff855247700 1 --2- 172.21.15.173:0/4120591622 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff83802ff60 0x7ff838032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:45.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:45.074+0000 7ff855247700 1 --2- 172.21.15.173:0/4120591622 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff8500f5410 0x7ff850109450 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:45.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:45.074+0000 7ff855247700 1 -- 172.21.15.173:0/4120591622 >> 172.21.15.173:0/4120591622 conn(0x7ff8500f0b70 msgr2=0x7ff8500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:45.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:45.074+0000 7ff855247700 1 -- 172.21.15.173:0/4120591622 shutdown_connections 2024-01-21T19:52:45.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:45.074+0000 7ff855247700 1 -- 172.21.15.173:0/4120591622 wait complete. 2024-01-21T19:52:45.067 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:52:45.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:44 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:42.972173+0000 mgr.smithi169.espgue (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:46.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:45 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:45.073065+0000 mon.smithi169 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.173:0/4120591622' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:52:46.514 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:52:46.514 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:52:47.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:46 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:44.972498+0000 mgr.smithi169.espgue (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:47.730 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.737+0000 7f4d6f157700 1 -- 172.21.15.173:0/1836865436 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d680f6a00 msgr2=0x7f4d680f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:47.730 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.737+0000 7f4d6f157700 1 --2- 172.21.15.173:0/1836865436 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d680f6a00 0x7f4d680f6e20 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f4d60009960 tx=0x7f4d60004d10).stop 2024-01-21T19:52:47.730 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.737+0000 7f4d6f157700 1 -- 172.21.15.173:0/1836865436 shutdown_connections 2024-01-21T19:52:47.730 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.737+0000 7f4d6f157700 1 --2- 172.21.15.173:0/1836865436 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d680f6a00 0x7f4d680f6e20 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:47.730 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.737+0000 7f4d6f157700 1 -- 172.21.15.173:0/1836865436 >> 172.21.15.173:0/1836865436 conn(0x7f4d680f02e0 msgr2=0x7f4d680f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:47.730 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.738+0000 7f4d6f157700 1 -- 172.21.15.173:0/1836865436 shutdown_connections 2024-01-21T19:52:47.730 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.738+0000 7f4d6f157700 1 -- 172.21.15.173:0/1836865436 wait complete. 2024-01-21T19:52:47.730 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.738+0000 7f4d6f157700 1 Processor -- start 2024-01-21T19:52:47.730 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.738+0000 7f4d6f157700 1 -- start start 2024-01-21T19:52:47.732 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.738+0000 7f4d6f157700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d680f6a00 0x7f4d681093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:47.732 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.738+0000 7f4d6f157700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d680f52f0 con 0x7f4d680f6a00 2024-01-21T19:52:47.732 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.739+0000 7f4d6cef3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d680f6a00 0x7f4d681093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:47.732 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.739+0000 7f4d6cef3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d680f6a00 0x7f4d681093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:54794/0 (socket says 172.21.15.173:54794) 2024-01-21T19:52:47.732 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.739+0000 7f4d6cef3700 1 -- 172.21.15.173:0/2713604047 learned_addr learned my addr 172.21.15.173:0/2713604047 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:52:47.732 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.739+0000 7f4d6cef3700 1 -- 172.21.15.173:0/2713604047 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d60009640 con 0x7f4d680f6a00 2024-01-21T19:52:47.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.739+0000 7f4d6cef3700 1 --2- 172.21.15.173:0/2713604047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d680f6a00 0x7f4d681093d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f4d60004b70 tx=0x7f4d60004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:47.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.739+0000 7f4d657fa700 1 -- 172.21.15.173:0/2713604047 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d60018070 con 0x7f4d680f6a00 2024-01-21T19:52:47.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.739+0000 7f4d6f157700 1 -- 172.21.15.173:0/2713604047 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d680f54f0 con 0x7f4d680f6a00 2024-01-21T19:52:47.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.739+0000 7f4d657fa700 1 -- 172.21.15.173:0/2713604047 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d6000b870 con 0x7f4d680f6a00 2024-01-21T19:52:47.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.739+0000 7f4d657fa700 1 -- 172.21.15.173:0/2713604047 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d6001c560 con 0x7f4d680f6a00 2024-01-21T19:52:47.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.739+0000 7f4d6f157700 1 -- 172.21.15.173:0/2713604047 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d68109a80 con 0x7f4d680f6a00 2024-01-21T19:52:47.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.740+0000 7f4d657fa700 1 -- 172.21.15.173:0/2713604047 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d60013630 con 0x7f4d680f6a00 2024-01-21T19:52:47.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.740+0000 7f4d6f157700 1 -- 172.21.15.173:0/2713604047 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d68045ad0 con 0x7f4d680f6a00 2024-01-21T19:52:47.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.740+0000 7f4d657fa700 1 --2- 172.21.15.173:0/2713604047 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4d5402ff60 0x7f4d54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:47.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.740+0000 7f4d657fa700 1 -- 172.21.15.173:0/2713604047 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d600411d0 con 0x7f4d680f6a00 2024-01-21T19:52:47.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.741+0000 7f4d67fff700 1 --2- 172.21.15.173:0/2713604047 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4d5402ff60 0x7f4d54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:47.734 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.741+0000 7f4d67fff700 1 --2- 172.21.15.173:0/2713604047 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4d5402ff60 0x7f4d54032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f4d500096a0 tx=0x7f4d50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:47.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.745+0000 7f4d657fa700 1 -- 172.21.15.173:0/2713604047 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d60004810 con 0x7f4d680f6a00 2024-01-21T19:52:47.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.949+0000 7f4d6f157700 1 -- 172.21.15.173:0/2713604047 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4d6810a2a0 con 0x7f4d680f6a00 2024-01-21T19:52:47.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.949+0000 7f4d657fa700 1 -- 172.21.15.173:0/2713604047 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4d60012380 con 0x7f4d680f6a00 2024-01-21T19:52:47.941 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:52:47.941 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:52:47.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.951+0000 7f4d6f157700 1 -- 172.21.15.173:0/2713604047 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4d5402ff60 msgr2=0x7f4d54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:47.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.951+0000 7f4d6f157700 1 --2- 172.21.15.173:0/2713604047 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4d5402ff60 0x7f4d54032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f4d500096a0 tx=0x7f4d50006b40).stop 2024-01-21T19:52:47.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.951+0000 7f4d6f157700 1 -- 172.21.15.173:0/2713604047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d680f6a00 msgr2=0x7f4d681093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:47.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.951+0000 7f4d6f157700 1 --2- 172.21.15.173:0/2713604047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d680f6a00 0x7f4d681093d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f4d60004b70 tx=0x7f4d60004d10).stop 2024-01-21T19:52:47.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.951+0000 7f4d6f157700 1 -- 172.21.15.173:0/2713604047 shutdown_connections 2024-01-21T19:52:47.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.951+0000 7f4d6f157700 1 --2- 172.21.15.173:0/2713604047 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4d5402ff60 0x7f4d54032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:47.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.951+0000 7f4d6f157700 1 --2- 172.21.15.173:0/2713604047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4d680f6a00 0x7f4d681093d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:47.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.951+0000 7f4d6f157700 1 -- 172.21.15.173:0/2713604047 >> 172.21.15.173:0/2713604047 conn(0x7f4d680f02e0 msgr2=0x7f4d680f0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:47.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.951+0000 7f4d6f157700 1 -- 172.21.15.173:0/2713604047 shutdown_connections 2024-01-21T19:52:47.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:47.951+0000 7f4d6f157700 1 -- 172.21.15.173:0/2713604047 wait complete. 2024-01-21T19:52:47.944 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:52:49.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:46.972897+0000 mgr.smithi169.espgue (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:49.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:47.950214+0000 mon.smithi169 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.173:0/2713604047' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:52:49.335 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:52:49.335 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:52:50.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.621+0000 7f29c677c700 1 -- 172.21.15.173:0/1832896140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29c00f6510 msgr2=0x7f29c00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:50.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.621+0000 7f29c677c700 1 --2- 172.21.15.173:0/1832896140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29c00f6510 0x7f29c00f6930 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f29b0009fb0 tx=0x7f29b0009b30).stop 2024-01-21T19:52:50.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.621+0000 7f29c677c700 1 -- 172.21.15.173:0/1832896140 shutdown_connections 2024-01-21T19:52:50.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.621+0000 7f29c677c700 1 --2- 172.21.15.173:0/1832896140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29c00f6510 0x7f29c00f6930 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:50.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.621+0000 7f29c677c700 1 -- 172.21.15.173:0/1832896140 >> 172.21.15.173:0/1832896140 conn(0x7f29c00f1c70 msgr2=0x7f29c00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:50.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.621+0000 7f29c677c700 1 -- 172.21.15.173:0/1832896140 shutdown_connections 2024-01-21T19:52:50.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.621+0000 7f29c677c700 1 -- 172.21.15.173:0/1832896140 wait complete. 2024-01-21T19:52:50.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.622+0000 7f29c677c700 1 Processor -- start 2024-01-21T19:52:50.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.622+0000 7f29c677c700 1 -- start start 2024-01-21T19:52:50.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.622+0000 7f29c677c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29c00f6510 0x7f29c006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:50.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.622+0000 7f29c677c700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c0069e20 con 0x7f29c00f6510 2024-01-21T19:52:50.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.622+0000 7f29bffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29c00f6510 0x7f29c006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:50.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.622+0000 7f29bffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29c00f6510 0x7f29c006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:54810/0 (socket says 172.21.15.173:54810) 2024-01-21T19:52:50.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.622+0000 7f29bffff700 1 -- 172.21.15.173:0/2241782849 learned_addr learned my addr 172.21.15.173:0/2241782849 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:52:50.616 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.623+0000 7f29bffff700 1 -- 172.21.15.173:0/2241782849 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b0005040 con 0x7f29c00f6510 2024-01-21T19:52:50.617 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.623+0000 7f29bffff700 1 --2- 172.21.15.173:0/2241782849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29c00f6510 0x7f29c006b9a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f29b0014000 tx=0x7f29b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:50.617 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.623+0000 7f29bcff9700 1 -- 172.21.15.173:0/2241782849 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b00095f0 con 0x7f29c00f6510 2024-01-21T19:52:50.617 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.623+0000 7f29c677c700 1 -- 172.21.15.173:0/2241782849 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c0069fc0 con 0x7f29c00f6510 2024-01-21T19:52:50.617 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.623+0000 7f29bcff9700 1 -- 172.21.15.173:0/2241782849 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29b0007e70 con 0x7f29c00f6510 2024-01-21T19:52:50.617 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.623+0000 7f29bcff9700 1 -- 172.21.15.173:0/2241782849 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b0014ea0 con 0x7f29c00f6510 2024-01-21T19:52:50.617 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.623+0000 7f29c677c700 1 -- 172.21.15.173:0/2241782849 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c006a450 con 0x7f29c00f6510 2024-01-21T19:52:50.617 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.624+0000 7f29bcff9700 1 -- 172.21.15.173:0/2241782849 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f29b0025070 con 0x7f29c00f6510 2024-01-21T19:52:50.617 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.624+0000 7f29c677c700 1 -- 172.21.15.173:0/2241782849 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c0059100 con 0x7f29c00f6510 2024-01-21T19:52:50.617 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.624+0000 7f29bcff9700 1 --2- 172.21.15.173:0/2241782849 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f29ac02ff60 0x7f29ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:50.617 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.624+0000 7f29bcff9700 1 -- 172.21.15.173:0/2241782849 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f29b0040a10 con 0x7f29c00f6510 2024-01-21T19:52:50.617 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.624+0000 7f29bf7fe700 1 --2- 172.21.15.173:0/2241782849 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f29ac02ff60 0x7f29ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:50.617 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.625+0000 7f29bf7fe700 1 --2- 172.21.15.173:0/2241782849 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f29ac02ff60 0x7f29ac032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f29a80096a0 tx=0x7f29a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:50.620 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.629+0000 7f29bcff9700 1 -- 172.21.15.173:0/2241782849 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29b000f4e0 con 0x7f29c00f6510 2024-01-21T19:52:50.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.830+0000 7f29c677c700 1 -- 172.21.15.173:0/2241782849 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29c0045ad0 con 0x7f29c00f6510 2024-01-21T19:52:50.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.831+0000 7f29bcff9700 1 -- 172.21.15.173:0/2241782849 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f29b000f300 con 0x7f29c00f6510 2024-01-21T19:52:50.824 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:52:50.824 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:52:50.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.833+0000 7f29c677c700 1 -- 172.21.15.173:0/2241782849 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f29ac02ff60 msgr2=0x7f29ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:50.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.833+0000 7f29c677c700 1 --2- 172.21.15.173:0/2241782849 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f29ac02ff60 0x7f29ac032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f29a80096a0 tx=0x7f29a8006b40).stop 2024-01-21T19:52:50.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.833+0000 7f29c677c700 1 -- 172.21.15.173:0/2241782849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29c00f6510 msgr2=0x7f29c006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:50.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.833+0000 7f29c677c700 1 --2- 172.21.15.173:0/2241782849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29c00f6510 0x7f29c006b9a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f29b0014000 tx=0x7f29b000b040).stop 2024-01-21T19:52:50.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.833+0000 7f29c677c700 1 -- 172.21.15.173:0/2241782849 shutdown_connections 2024-01-21T19:52:50.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.833+0000 7f29c677c700 1 --2- 172.21.15.173:0/2241782849 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f29ac02ff60 0x7f29ac032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:50.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.833+0000 7f29c677c700 1 --2- 172.21.15.173:0/2241782849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f29c00f6510 0x7f29c006b9a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:50.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.833+0000 7f29c677c700 1 -- 172.21.15.173:0/2241782849 >> 172.21.15.173:0/2241782849 conn(0x7f29c00f1c70 msgr2=0x7f29c00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:50.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.833+0000 7f29c677c700 1 -- 172.21.15.173:0/2241782849 shutdown_connections 2024-01-21T19:52:50.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:50.833+0000 7f29c677c700 1 -- 172.21.15.173:0/2241782849 wait complete. 2024-01-21T19:52:50.827 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:52:51.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:50 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:48.973235+0000 mgr.smithi169.espgue (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:52.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:51 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:50.832883+0000 mon.smithi169 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.173:0/2241782849' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:52:52.263 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:52:52.264 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:52:53.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:52 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:50.973465+0000 mgr.smithi169.espgue (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:53.523 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.529+0000 7f8d5619e700 1 -- 172.21.15.173:0/3622438449 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d500f51a0 msgr2=0x7f8d500f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:53.523 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.529+0000 7f8d5619e700 1 --2- 172.21.15.173:0/3622438449 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d500f51a0 0x7f8d500f55c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f8d40009fb0 tx=0x7f8d40009b30).stop 2024-01-21T19:52:53.523 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.529+0000 7f8d5619e700 1 -- 172.21.15.173:0/3622438449 shutdown_connections 2024-01-21T19:52:53.524 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.529+0000 7f8d5619e700 1 --2- 172.21.15.173:0/3622438449 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d500f51a0 0x7f8d500f55c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:53.524 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.529+0000 7f8d5619e700 1 -- 172.21.15.173:0/3622438449 >> 172.21.15.173:0/3622438449 conn(0x7f8d500f09a0 msgr2=0x7f8d500f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:53.524 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.529+0000 7f8d5619e700 1 -- 172.21.15.173:0/3622438449 shutdown_connections 2024-01-21T19:52:53.524 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.529+0000 7f8d5619e700 1 -- 172.21.15.173:0/3622438449 wait complete. 2024-01-21T19:52:53.524 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.529+0000 7f8d5619e700 1 Processor -- start 2024-01-21T19:52:53.524 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.529+0000 7f8d5619e700 1 -- start start 2024-01-21T19:52:53.524 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.530+0000 7f8d5619e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d500f51a0 0x7f8d500fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:53.524 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.530+0000 7f8d5619e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d500fb5a0 con 0x7f8d500f51a0 2024-01-21T19:52:53.524 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.530+0000 7f8d5519c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d500f51a0 0x7f8d500fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:53.524 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.530+0000 7f8d5519c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d500f51a0 0x7f8d500fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:37830/0 (socket says 172.21.15.173:37830) 2024-01-21T19:52:53.524 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.530+0000 7f8d5519c700 1 -- 172.21.15.173:0/164632323 learned_addr learned my addr 172.21.15.173:0/164632323 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:52:53.524 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.530+0000 7f8d5519c700 1 -- 172.21.15.173:0/164632323 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d40005040 con 0x7f8d500f51a0 2024-01-21T19:52:53.524 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.530+0000 7f8d5519c700 1 --2- 172.21.15.173:0/164632323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d500f51a0 0x7f8d500fd120 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f8d40013510 tx=0x7f8d4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:53.524 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.530+0000 7f8d45ffb700 1 -- 172.21.15.173:0/164632323 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d40009320 con 0x7f8d500f51a0 2024-01-21T19:52:53.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.530+0000 7f8d5619e700 1 -- 172.21.15.173:0/164632323 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d500fb740 con 0x7f8d500f51a0 2024-01-21T19:52:53.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.530+0000 7f8d45ffb700 1 -- 172.21.15.173:0/164632323 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d40013f00 con 0x7f8d500f51a0 2024-01-21T19:52:53.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.530+0000 7f8d45ffb700 1 -- 172.21.15.173:0/164632323 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d40014dc0 con 0x7f8d500f51a0 2024-01-21T19:52:53.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.531+0000 7f8d5619e700 1 -- 172.21.15.173:0/164632323 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d500fbbd0 con 0x7f8d500f51a0 2024-01-21T19:52:53.526 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.531+0000 7f8d45ffb700 1 -- 172.21.15.173:0/164632323 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8d4001d3d0 con 0x7f8d500f51a0 2024-01-21T19:52:53.526 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.532+0000 7f8d45ffb700 1 --2- 172.21.15.173:0/164632323 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8d3c02ff60 0x7f8d3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:53.526 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.532+0000 7f8d45ffb700 1 -- 172.21.15.173:0/164632323 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8d40041100 con 0x7f8d500f51a0 2024-01-21T19:52:53.526 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.532+0000 7f8d5619e700 1 -- 172.21.15.173:0/164632323 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d50059100 con 0x7f8d500f51a0 2024-01-21T19:52:53.526 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.532+0000 7f8d5499b700 1 --2- 172.21.15.173:0/164632323 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8d3c02ff60 0x7f8d3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:53.526 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.532+0000 7f8d5499b700 1 --2- 172.21.15.173:0/164632323 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8d3c02ff60 0x7f8d3c032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f8d4c0096a0 tx=0x7f8d4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:53.529 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.537+0000 7f8d45ffb700 1 -- 172.21.15.173:0/164632323 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d40012920 con 0x7f8d500f51a0 2024-01-21T19:52:53.732 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.740+0000 7f8d5619e700 1 -- 172.21.15.173:0/164632323 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d500fc2d0 con 0x7f8d500f51a0 2024-01-21T19:52:53.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.740+0000 7f8d45ffb700 1 -- 172.21.15.173:0/164632323 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8d40012330 con 0x7f8d500f51a0 2024-01-21T19:52:53.733 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:52:53.733 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:52:53.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.742+0000 7f8d5619e700 1 -- 172.21.15.173:0/164632323 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8d3c02ff60 msgr2=0x7f8d3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:53.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.742+0000 7f8d5619e700 1 --2- 172.21.15.173:0/164632323 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8d3c02ff60 0x7f8d3c032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f8d4c0096a0 tx=0x7f8d4c006b40).stop 2024-01-21T19:52:53.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.742+0000 7f8d5619e700 1 -- 172.21.15.173:0/164632323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d500f51a0 msgr2=0x7f8d500fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:53.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.742+0000 7f8d5619e700 1 --2- 172.21.15.173:0/164632323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d500f51a0 0x7f8d500fd120 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f8d40013510 tx=0x7f8d4000b040).stop 2024-01-21T19:52:53.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.742+0000 7f8d5619e700 1 -- 172.21.15.173:0/164632323 shutdown_connections 2024-01-21T19:52:53.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.742+0000 7f8d5619e700 1 --2- 172.21.15.173:0/164632323 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8d3c02ff60 0x7f8d3c032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:53.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.742+0000 7f8d5619e700 1 --2- 172.21.15.173:0/164632323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d500f51a0 0x7f8d500fd120 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:53.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.742+0000 7f8d5619e700 1 -- 172.21.15.173:0/164632323 >> 172.21.15.173:0/164632323 conn(0x7f8d500f09a0 msgr2=0x7f8d5005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:53.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.742+0000 7f8d5619e700 1 -- 172.21.15.173:0/164632323 shutdown_connections 2024-01-21T19:52:53.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:53.742+0000 7f8d5619e700 1 -- 172.21.15.173:0/164632323 wait complete. 2024-01-21T19:52:53.736 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:52:54.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:53 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:53.741896+0000 mon.smithi169 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.173:0/164632323' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:52:55.144 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:52:55.145 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:52:55.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:54 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:52.973806+0000 mgr.smithi169.espgue (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:56.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.414+0000 7f0b8b59e700 1 -- 172.21.15.173:0/1096922104 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b8c0f62a0 msgr2=0x7f0b8c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:56.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.414+0000 7f0b8b59e700 1 --2- 172.21.15.173:0/1096922104 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b8c0f62a0 0x7f0b8c0f66c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f0b7c009fb0 tx=0x7f0b7c009b30).stop 2024-01-21T19:52:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.415+0000 7f0b8b59e700 1 -- 172.21.15.173:0/1096922104 shutdown_connections 2024-01-21T19:52:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.415+0000 7f0b8b59e700 1 --2- 172.21.15.173:0/1096922104 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b8c0f62a0 0x7f0b8c0f66c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.415+0000 7f0b8b59e700 1 -- 172.21.15.173:0/1096922104 >> 172.21.15.173:0/1096922104 conn(0x7f0b8c0f1aa0 msgr2=0x7f0b8c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.415+0000 7f0b8b59e700 1 -- 172.21.15.173:0/1096922104 shutdown_connections 2024-01-21T19:52:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.415+0000 7f0b8b59e700 1 -- 172.21.15.173:0/1096922104 wait complete. 2024-01-21T19:52:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.415+0000 7f0b8b59e700 1 Processor -- start 2024-01-21T19:52:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.415+0000 7f0b8b59e700 1 -- start start 2024-01-21T19:52:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.415+0000 7f0b8b59e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b8c0f62a0 0x7f0b8c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.415+0000 7f0b8b59e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8c1069a0 con 0x7f0b8c0f62a0 2024-01-21T19:52:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.416+0000 7f0b8a59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b8c0f62a0 0x7f0b8c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.416+0000 7f0b8a59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b8c0f62a0 0x7f0b8c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:37840/0 (socket says 172.21.15.173:37840) 2024-01-21T19:52:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.416+0000 7f0b8a59c700 1 -- 172.21.15.173:0/2228545005 learned_addr learned my addr 172.21.15.173:0/2228545005 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:52:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.416+0000 7f0b8a59c700 1 -- 172.21.15.173:0/2228545005 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c005040 con 0x7f0b8c0f62a0 2024-01-21T19:52:56.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.416+0000 7f0b8a59c700 1 --2- 172.21.15.173:0/2228545005 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b8c0f62a0 0x7f0b8c1064b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f0b7c013510 tx=0x7f0b7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:56.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.416+0000 7f0b82ffd700 1 -- 172.21.15.173:0/2228545005 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b7c009320 con 0x7f0b8c0f62a0 2024-01-21T19:52:56.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.416+0000 7f0b8b59e700 1 -- 172.21.15.173:0/2228545005 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b8c106ba0 con 0x7f0b8c0f62a0 2024-01-21T19:52:56.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.416+0000 7f0b82ffd700 1 -- 172.21.15.173:0/2228545005 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b7c013f00 con 0x7f0b8c0f62a0 2024-01-21T19:52:56.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.416+0000 7f0b82ffd700 1 -- 172.21.15.173:0/2228545005 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b7c014dc0 con 0x7f0b8c0f62a0 2024-01-21T19:52:56.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.416+0000 7f0b8b59e700 1 -- 172.21.15.173:0/2228545005 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b8c107030 con 0x7f0b8c0f62a0 2024-01-21T19:52:56.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.417+0000 7f0b82ffd700 1 -- 172.21.15.173:0/2228545005 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0b7c01d3d0 con 0x7f0b8c0f62a0 2024-01-21T19:52:56.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.417+0000 7f0b8b59e700 1 -- 172.21.15.173:0/2228545005 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b8c045ad0 con 0x7f0b8c0f62a0 2024-01-21T19:52:56.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.417+0000 7f0b82ffd700 1 --2- 172.21.15.173:0/2228545005 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0b7002ff60 0x7f0b70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:56.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.417+0000 7f0b82ffd700 1 -- 172.21.15.173:0/2228545005 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0b7c041300 con 0x7f0b8c0f62a0 2024-01-21T19:52:56.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.418+0000 7f0b89d9b700 1 --2- 172.21.15.173:0/2228545005 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0b7002ff60 0x7f0b70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:56.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.418+0000 7f0b89d9b700 1 --2- 172.21.15.173:0/2228545005 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0b7002ff60 0x7f0b70032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f0b780096a0 tx=0x7f0b78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:56.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.422+0000 7f0b82ffd700 1 -- 172.21.15.173:0/2228545005 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b7c0263a0 con 0x7f0b8c0f62a0 2024-01-21T19:52:56.619 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.626+0000 7f0b8b59e700 1 -- 172.21.15.173:0/2228545005 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b8c107ee0 con 0x7f0b8c0f62a0 2024-01-21T19:52:56.619 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.627+0000 7f0b82ffd700 1 -- 172.21.15.173:0/2228545005 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0b7c01cb10 con 0x7f0b8c0f62a0 2024-01-21T19:52:56.620 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:52:56.620 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:52:56.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.628+0000 7f0b8b59e700 1 -- 172.21.15.173:0/2228545005 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0b7002ff60 msgr2=0x7f0b70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:56.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.628+0000 7f0b8b59e700 1 --2- 172.21.15.173:0/2228545005 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0b7002ff60 0x7f0b70032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f0b780096a0 tx=0x7f0b78006b40).stop 2024-01-21T19:52:56.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.628+0000 7f0b8b59e700 1 -- 172.21.15.173:0/2228545005 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b8c0f62a0 msgr2=0x7f0b8c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:56.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.628+0000 7f0b8b59e700 1 --2- 172.21.15.173:0/2228545005 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b8c0f62a0 0x7f0b8c1064b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f0b7c013510 tx=0x7f0b7c00b040).stop 2024-01-21T19:52:56.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.628+0000 7f0b8b59e700 1 -- 172.21.15.173:0/2228545005 shutdown_connections 2024-01-21T19:52:56.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.628+0000 7f0b8b59e700 1 --2- 172.21.15.173:0/2228545005 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0b7002ff60 0x7f0b70032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:56.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.628+0000 7f0b8b59e700 1 --2- 172.21.15.173:0/2228545005 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0b8c0f62a0 0x7f0b8c1064b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:56.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.628+0000 7f0b8b59e700 1 -- 172.21.15.173:0/2228545005 >> 172.21.15.173:0/2228545005 conn(0x7f0b8c0f1aa0 msgr2=0x7f0b8c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:56.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.628+0000 7f0b8b59e700 1 -- 172.21.15.173:0/2228545005 shutdown_connections 2024-01-21T19:52:56.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:56.628+0000 7f0b8b59e700 1 -- 172.21.15.173:0/2228545005 wait complete. 2024-01-21T19:52:56.622 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:52:57.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:56 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:54.974160+0000 mgr.smithi169.espgue (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:57.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:56 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:56.628218+0000 mon.smithi169 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.173:0/2228545005' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:52:58.027 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:52:58.028 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:52:59.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:56.974418+0000 mgr.smithi169.espgue (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:52:59.221 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.227+0000 7f1fa3621700 1 -- 172.21.15.173:0/3234031324 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f9c060280 msgr2=0x7f1f9c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:59.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.227+0000 7f1fa3621700 1 --2- 172.21.15.173:0/3234031324 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f9c060280 0x7f1f9c0f90f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f1f98009fb0 tx=0x7f1f98009b30).stop 2024-01-21T19:52:59.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.227+0000 7f1fa3621700 1 -- 172.21.15.173:0/3234031324 shutdown_connections 2024-01-21T19:52:59.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.227+0000 7f1fa3621700 1 --2- 172.21.15.173:0/3234031324 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f9c060280 0x7f1f9c0f90f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:59.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.227+0000 7f1fa3621700 1 -- 172.21.15.173:0/3234031324 >> 172.21.15.173:0/3234031324 conn(0x7f1f9c0f0b70 msgr2=0x7f1f9c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:59.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.227+0000 7f1fa3621700 1 -- 172.21.15.173:0/3234031324 shutdown_connections 2024-01-21T19:52:59.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.227+0000 7f1fa3621700 1 -- 172.21.15.173:0/3234031324 wait complete. 2024-01-21T19:52:59.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.228+0000 7f1fa3621700 1 Processor -- start 2024-01-21T19:52:59.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.228+0000 7f1fa3621700 1 -- start start 2024-01-21T19:52:59.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.228+0000 7f1fa3621700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f9c060280 0x7f1f9c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:59.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.228+0000 7f1fa3621700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c0f4ff0 con 0x7f1f9c060280 2024-01-21T19:52:59.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.228+0000 7f1fa13bd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f9c060280 0x7f1f9c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:59.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.228+0000 7f1fa13bd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f9c060280 0x7f1f9c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:37850/0 (socket says 172.21.15.173:37850) 2024-01-21T19:52:59.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.228+0000 7f1fa13bd700 1 -- 172.21.15.173:0/532986870 learned_addr learned my addr 172.21.15.173:0/532986870 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:52:59.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.228+0000 7f1fa13bd700 1 -- 172.21.15.173:0/532986870 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f98005040 con 0x7f1f9c060280 2024-01-21T19:52:59.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.228+0000 7f1fa13bd700 1 --2- 172.21.15.173:0/532986870 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f9c060280 0x7f1f9c0f6b70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f1f98013510 tx=0x7f1f9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:59.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.228+0000 7f1f91ffb700 1 -- 172.21.15.173:0/532986870 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f98009320 con 0x7f1f9c060280 2024-01-21T19:52:59.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.229+0000 7f1fa3621700 1 -- 172.21.15.173:0/532986870 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f9c0f5190 con 0x7f1f9c060280 2024-01-21T19:52:59.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.229+0000 7f1f91ffb700 1 -- 172.21.15.173:0/532986870 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f98013f00 con 0x7f1f9c060280 2024-01-21T19:52:59.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.229+0000 7f1f91ffb700 1 -- 172.21.15.173:0/532986870 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f98014dc0 con 0x7f1f9c060280 2024-01-21T19:52:59.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.229+0000 7f1fa3621700 1 -- 172.21.15.173:0/532986870 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f9c0f5620 con 0x7f1f9c060280 2024-01-21T19:52:59.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.229+0000 7f1f91ffb700 1 -- 172.21.15.173:0/532986870 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1f980095f0 con 0x7f1f9c060280 2024-01-21T19:52:59.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.230+0000 7f1fa3621700 1 -- 172.21.15.173:0/532986870 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f9c059100 con 0x7f1f9c060280 2024-01-21T19:52:59.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.230+0000 7f1f91ffb700 1 --2- 172.21.15.173:0/532986870 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1f8802ff60 0x7f1f88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:52:59.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.230+0000 7f1f91ffb700 1 -- 172.21.15.173:0/532986870 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1f980410a0 con 0x7f1f9c060280 2024-01-21T19:52:59.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.230+0000 7f1fa0bbc700 1 --2- 172.21.15.173:0/532986870 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1f8802ff60 0x7f1f88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:52:59.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.230+0000 7f1fa0bbc700 1 --2- 172.21.15.173:0/532986870 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1f8802ff60 0x7f1f88032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f1f8c0096a0 tx=0x7f1f8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:52:59.227 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.234+0000 7f1f91ffb700 1 -- 172.21.15.173:0/532986870 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f98014420 con 0x7f1f9c060280 2024-01-21T19:52:59.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.439+0000 7f1fa3621700 1 -- 172.21.15.173:0/532986870 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f9c045ad0 con 0x7f1f9c060280 2024-01-21T19:52:59.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.439+0000 7f1f91ffb700 1 -- 172.21.15.173:0/532986870 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1f9801c920 con 0x7f1f9c060280 2024-01-21T19:52:59.432 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:52:59.433 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:52:59.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.441+0000 7f1fa3621700 1 -- 172.21.15.173:0/532986870 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1f8802ff60 msgr2=0x7f1f88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:59.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.441+0000 7f1fa3621700 1 --2- 172.21.15.173:0/532986870 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1f8802ff60 0x7f1f88032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f1f8c0096a0 tx=0x7f1f8c006b40).stop 2024-01-21T19:52:59.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.441+0000 7f1fa3621700 1 -- 172.21.15.173:0/532986870 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f9c060280 msgr2=0x7f1f9c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:52:59.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.441+0000 7f1fa3621700 1 --2- 172.21.15.173:0/532986870 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f9c060280 0x7f1f9c0f6b70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f1f98013510 tx=0x7f1f9800b040).stop 2024-01-21T19:52:59.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.441+0000 7f1fa3621700 1 -- 172.21.15.173:0/532986870 shutdown_connections 2024-01-21T19:52:59.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.441+0000 7f1fa3621700 1 --2- 172.21.15.173:0/532986870 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f1f8802ff60 0x7f1f88032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:59.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.441+0000 7f1fa3621700 1 --2- 172.21.15.173:0/532986870 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1f9c060280 0x7f1f9c0f6b70 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:52:59.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.441+0000 7f1fa3621700 1 -- 172.21.15.173:0/532986870 >> 172.21.15.173:0/532986870 conn(0x7f1f9c0f0b70 msgr2=0x7f1f9c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:52:59.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.441+0000 7f1fa3621700 1 -- 172.21.15.173:0/532986870 shutdown_connections 2024-01-21T19:52:59.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:52:59.441+0000 7f1fa3621700 1 -- 172.21.15.173:0/532986870 wait complete. 2024-01-21T19:52:59.435 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:00.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:52:59 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:52:59.441121+0000 mon.smithi169 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.173:0/532986870' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:53:00.831 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:53:00.831 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:53:01.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:00 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:52:58.974706+0000 mgr.smithi169.espgue (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:02.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.044+0000 7fe68ac43700 1 -- 172.21.15.173:0/172521471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6840f5410 msgr2=0x7fe6840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:02.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.044+0000 7fe68ac43700 1 --2- 172.21.15.173:0/172521471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6840f5410 0x7fe6840f5830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fe678009fb0 tx=0x7fe678009b30).stop 2024-01-21T19:53:02.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.044+0000 7fe68ac43700 1 -- 172.21.15.173:0/172521471 shutdown_connections 2024-01-21T19:53:02.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.044+0000 7fe68ac43700 1 --2- 172.21.15.173:0/172521471 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6840f5410 0x7fe6840f5830 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:02.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.044+0000 7fe68ac43700 1 -- 172.21.15.173:0/172521471 >> 172.21.15.173:0/172521471 conn(0x7fe6840f0b70 msgr2=0x7fe6840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:02.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.044+0000 7fe68ac43700 1 -- 172.21.15.173:0/172521471 shutdown_connections 2024-01-21T19:53:02.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.044+0000 7fe68ac43700 1 -- 172.21.15.173:0/172521471 wait complete. 2024-01-21T19:53:02.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.044+0000 7fe68ac43700 1 Processor -- start 2024-01-21T19:53:02.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.044+0000 7fe68ac43700 1 -- start start 2024-01-21T19:53:02.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.045+0000 7fe68ac43700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6840f5410 0x7fe684105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:02.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.045+0000 7fe68ac43700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe684105b20 con 0x7fe6840f5410 2024-01-21T19:53:02.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.045+0000 7fe6889df700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6840f5410 0x7fe684105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:02.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.045+0000 7fe6889df700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6840f5410 0x7fe684105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:37864/0 (socket says 172.21.15.173:37864) 2024-01-21T19:53:02.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.045+0000 7fe6889df700 1 -- 172.21.15.173:0/1889932031 learned_addr learned my addr 172.21.15.173:0/1889932031 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:53:02.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.045+0000 7fe6889df700 1 -- 172.21.15.173:0/1889932031 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe678005040 con 0x7fe6840f5410 2024-01-21T19:53:02.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.045+0000 7fe6889df700 1 --2- 172.21.15.173:0/1889932031 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6840f5410 0x7fe684105630 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fe678013510 tx=0x7fe67800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:02.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.045+0000 7fe6817fa700 1 -- 172.21.15.173:0/1889932031 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe678009320 con 0x7fe6840f5410 2024-01-21T19:53:02.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.045+0000 7fe68ac43700 1 -- 172.21.15.173:0/1889932031 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe684105d20 con 0x7fe6840f5410 2024-01-21T19:53:02.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.045+0000 7fe6817fa700 1 -- 172.21.15.173:0/1889932031 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe678013f00 con 0x7fe6840f5410 2024-01-21T19:53:02.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.045+0000 7fe6817fa700 1 -- 172.21.15.173:0/1889932031 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe678014d90 con 0x7fe6840f5410 2024-01-21T19:53:02.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.045+0000 7fe68ac43700 1 -- 172.21.15.173:0/1889932031 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6841061b0 con 0x7fe6840f5410 2024-01-21T19:53:02.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.046+0000 7fe6817fa700 1 -- 172.21.15.173:0/1889932031 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe6780263a0 con 0x7fe6840f5410 2024-01-21T19:53:02.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.046+0000 7fe6817fa700 1 --2- 172.21.15.173:0/1889932031 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe67402ff60 0x7fe674032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:02.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.046+0000 7fe6817fa700 1 -- 172.21.15.173:0/1889932031 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe678041160 con 0x7fe6840f5410 2024-01-21T19:53:02.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.046+0000 7fe68ac43700 1 -- 172.21.15.173:0/1889932031 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe684059100 con 0x7fe6840f5410 2024-01-21T19:53:02.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.046+0000 7fe683fff700 1 --2- 172.21.15.173:0/1889932031 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe67402ff60 0x7fe674032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:02.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.047+0000 7fe683fff700 1 --2- 172.21.15.173:0/1889932031 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe67402ff60 0x7fe674032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fe6700096a0 tx=0x7fe670006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:02.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.051+0000 7fe6817fa700 1 -- 172.21.15.173:0/1889932031 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe678012920 con 0x7fe6840f5410 2024-01-21T19:53:02.248 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.255+0000 7fe68ac43700 1 -- 172.21.15.173:0/1889932031 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6840fae60 con 0x7fe6840f5410 2024-01-21T19:53:02.249 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.256+0000 7fe6817fa700 1 -- 172.21.15.173:0/1889932031 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe67802b020 con 0x7fe6840f5410 2024-01-21T19:53:02.249 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:53:02.249 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:53:02.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.258+0000 7fe68ac43700 1 -- 172.21.15.173:0/1889932031 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe67402ff60 msgr2=0x7fe674032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:02.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.258+0000 7fe68ac43700 1 --2- 172.21.15.173:0/1889932031 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe67402ff60 0x7fe674032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fe6700096a0 tx=0x7fe670006b40).stop 2024-01-21T19:53:02.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.258+0000 7fe68ac43700 1 -- 172.21.15.173:0/1889932031 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6840f5410 msgr2=0x7fe684105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:02.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.258+0000 7fe68ac43700 1 --2- 172.21.15.173:0/1889932031 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6840f5410 0x7fe684105630 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fe678013510 tx=0x7fe67800b040).stop 2024-01-21T19:53:02.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.258+0000 7fe68ac43700 1 -- 172.21.15.173:0/1889932031 shutdown_connections 2024-01-21T19:53:02.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.258+0000 7fe68ac43700 1 --2- 172.21.15.173:0/1889932031 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe67402ff60 0x7fe674032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:02.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.258+0000 7fe68ac43700 1 --2- 172.21.15.173:0/1889932031 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6840f5410 0x7fe684105630 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:02.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.258+0000 7fe68ac43700 1 -- 172.21.15.173:0/1889932031 >> 172.21.15.173:0/1889932031 conn(0x7fe6840f0b70 msgr2=0x7fe6840f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:02.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.258+0000 7fe68ac43700 1 -- 172.21.15.173:0/1889932031 shutdown_connections 2024-01-21T19:53:02.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:02.258+0000 7fe68ac43700 1 -- 172.21.15.173:0/1889932031 wait complete. 2024-01-21T19:53:02.252 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:03.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:02 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:00.975076+0000 mgr.smithi169.espgue (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:03.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:02 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:02.257934+0000 mon.smithi169 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.173:0/1889932031' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:53:03.696 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:53:03.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:53:04.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.872+0000 7fb0829d9700 1 -- 172.21.15.173:0/2120136084 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb07c0f5410 msgr2=0x7fb07c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:04.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.872+0000 7fb0829d9700 1 --2- 172.21.15.173:0/2120136084 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb07c0f5410 0x7fb07c0f5830 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fb06c009fb0 tx=0x7fb06c009b30).stop 2024-01-21T19:53:04.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.872+0000 7fb0829d9700 1 -- 172.21.15.173:0/2120136084 shutdown_connections 2024-01-21T19:53:04.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.872+0000 7fb0829d9700 1 --2- 172.21.15.173:0/2120136084 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb07c0f5410 0x7fb07c0f5830 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:04.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.872+0000 7fb0829d9700 1 -- 172.21.15.173:0/2120136084 >> 172.21.15.173:0/2120136084 conn(0x7fb07c0f0b70 msgr2=0x7fb07c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:04.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.872+0000 7fb0829d9700 1 -- 172.21.15.173:0/2120136084 shutdown_connections 2024-01-21T19:53:04.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.872+0000 7fb0829d9700 1 -- 172.21.15.173:0/2120136084 wait complete. 2024-01-21T19:53:04.867 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.873+0000 7fb0829d9700 1 Processor -- start 2024-01-21T19:53:04.867 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.873+0000 7fb0829d9700 1 -- start start 2024-01-21T19:53:04.867 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.874+0000 7fb0829d9700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb07c0f5410 0x7fb07c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:04.868 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.874+0000 7fb0829d9700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c060240 con 0x7fb07c0f5410 2024-01-21T19:53:04.868 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.874+0000 7fb07bfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb07c0f5410 0x7fb07c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:04.868 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.874+0000 7fb07bfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb07c0f5410 0x7fb07c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:34536/0 (socket says 172.21.15.173:34536) 2024-01-21T19:53:04.868 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.874+0000 7fb07bfff700 1 -- 172.21.15.173:0/627243061 learned_addr learned my addr 172.21.15.173:0/627243061 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:53:04.868 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.874+0000 7fb07bfff700 1 -- 172.21.15.173:0/627243061 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06c005040 con 0x7fb07c0f5410 2024-01-21T19:53:04.868 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.874+0000 7fb07bfff700 1 --2- 172.21.15.173:0/627243061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb07c0f5410 0x7fb07c109430 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fb06c014000 tx=0x7fb06c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:04.868 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.874+0000 7fb078ff9700 1 -- 172.21.15.173:0/627243061 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb06c0095f0 con 0x7fb07c0f5410 2024-01-21T19:53:04.868 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.875+0000 7fb0829d9700 1 -- 172.21.15.173:0/627243061 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb07c060440 con 0x7fb07c0f5410 2024-01-21T19:53:04.868 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.875+0000 7fb078ff9700 1 -- 172.21.15.173:0/627243061 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb06c007e70 con 0x7fb07c0f5410 2024-01-21T19:53:04.868 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.875+0000 7fb078ff9700 1 -- 172.21.15.173:0/627243061 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb06c014ea0 con 0x7fb07c0f5410 2024-01-21T19:53:04.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.875+0000 7fb0829d9700 1 -- 172.21.15.173:0/627243061 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb07c109ae0 con 0x7fb07c0f5410 2024-01-21T19:53:04.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.875+0000 7fb078ff9700 1 -- 172.21.15.173:0/627243061 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb06c025070 con 0x7fb07c0f5410 2024-01-21T19:53:04.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.875+0000 7fb0829d9700 1 -- 172.21.15.173:0/627243061 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb07c045ad0 con 0x7fb07c0f5410 2024-01-21T19:53:04.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.876+0000 7fb078ff9700 1 --2- 172.21.15.173:0/627243061 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb06402ff60 0x7fb064032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:04.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.876+0000 7fb078ff9700 1 -- 172.21.15.173:0/627243061 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb06c040920 con 0x7fb07c0f5410 2024-01-21T19:53:04.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.876+0000 7fb07b7fe700 1 --2- 172.21.15.173:0/627243061 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb06402ff60 0x7fb064032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:04.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.876+0000 7fb07b7fe700 1 --2- 172.21.15.173:0/627243061 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb06402ff60 0x7fb064032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fb0700096a0 tx=0x7fb070006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:04.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:04.881+0000 7fb078ff9700 1 -- 172.21.15.173:0/627243061 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb06c00f4e0 con 0x7fb07c0f5410 2024-01-21T19:53:05.079 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:05.087+0000 7fb0829d9700 1 -- 172.21.15.173:0/627243061 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb07c0fae60 con 0x7fb07c0f5410 2024-01-21T19:53:05.080 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:05.087+0000 7fb078ff9700 1 -- 172.21.15.173:0/627243061 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb06c00f300 con 0x7fb07c0f5410 2024-01-21T19:53:05.080 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:53:05.080 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:53:05.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:05.089+0000 7fb0829d9700 1 -- 172.21.15.173:0/627243061 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb06402ff60 msgr2=0x7fb064032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:05.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:05.089+0000 7fb0829d9700 1 --2- 172.21.15.173:0/627243061 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb06402ff60 0x7fb064032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fb0700096a0 tx=0x7fb070006b40).stop 2024-01-21T19:53:05.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:05.089+0000 7fb0829d9700 1 -- 172.21.15.173:0/627243061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb07c0f5410 msgr2=0x7fb07c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:05.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:05.089+0000 7fb0829d9700 1 --2- 172.21.15.173:0/627243061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb07c0f5410 0x7fb07c109430 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fb06c014000 tx=0x7fb06c00b040).stop 2024-01-21T19:53:05.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:05.089+0000 7fb0829d9700 1 -- 172.21.15.173:0/627243061 shutdown_connections 2024-01-21T19:53:05.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:05.089+0000 7fb0829d9700 1 --2- 172.21.15.173:0/627243061 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb06402ff60 0x7fb064032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:05.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:05.089+0000 7fb0829d9700 1 --2- 172.21.15.173:0/627243061 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb07c0f5410 0x7fb07c109430 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:05.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:05.089+0000 7fb0829d9700 1 -- 172.21.15.173:0/627243061 >> 172.21.15.173:0/627243061 conn(0x7fb07c0f0b70 msgr2=0x7fb07c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:05.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:05.089+0000 7fb0829d9700 1 -- 172.21.15.173:0/627243061 shutdown_connections 2024-01-21T19:53:05.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:05.089+0000 7fb0829d9700 1 -- 172.21.15.173:0/627243061 wait complete. 2024-01-21T19:53:05.083 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:05.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:04 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:02.975338+0000 mgr.smithi169.espgue (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:06.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:05 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:05.088873+0000 mon.smithi169 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.173:0/627243061' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:53:06.476 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:53:06.476 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:53:07.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:06 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:04.975666+0000 mgr.smithi169.espgue (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:07.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.714+0000 7f7528a4b700 1 -- 172.21.15.173:0/2444596688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75240f62a0 msgr2=0x7f75240f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:07.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.714+0000 7f7528a4b700 1 --2- 172.21.15.173:0/2444596688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75240f62a0 0x7f75240f66c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f7514009fb0 tx=0x7f7514009b30).stop 2024-01-21T19:53:07.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.714+0000 7f7528a4b700 1 -- 172.21.15.173:0/2444596688 shutdown_connections 2024-01-21T19:53:07.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.714+0000 7f7528a4b700 1 --2- 172.21.15.173:0/2444596688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75240f62a0 0x7f75240f66c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:07.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.714+0000 7f7528a4b700 1 -- 172.21.15.173:0/2444596688 >> 172.21.15.173:0/2444596688 conn(0x7f75240f1aa0 msgr2=0x7f75240f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:07.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.715+0000 7f7528a4b700 1 -- 172.21.15.173:0/2444596688 shutdown_connections 2024-01-21T19:53:07.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.715+0000 7f7528a4b700 1 -- 172.21.15.173:0/2444596688 wait complete. 2024-01-21T19:53:07.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.715+0000 7f7528a4b700 1 Processor -- start 2024-01-21T19:53:07.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.715+0000 7f7528a4b700 1 -- start start 2024-01-21T19:53:07.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.716+0000 7f7528a4b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75240f62a0 0x7f75241064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:07.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.716+0000 7f7528a4b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75241069a0 con 0x7f75240f62a0 2024-01-21T19:53:07.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.716+0000 7f75237fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75240f62a0 0x7f75241064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:07.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.716+0000 7f75237fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75240f62a0 0x7f75241064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:34560/0 (socket says 172.21.15.173:34560) 2024-01-21T19:53:07.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.716+0000 7f75237fe700 1 -- 172.21.15.173:0/3306092283 learned_addr learned my addr 172.21.15.173:0/3306092283 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:53:07.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.716+0000 7f75237fe700 1 -- 172.21.15.173:0/3306092283 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7514005040 con 0x7f75240f62a0 2024-01-21T19:53:07.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.716+0000 7f75237fe700 1 --2- 172.21.15.173:0/3306092283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75240f62a0 0x7f75241064b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f7514013510 tx=0x7f751400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:07.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.716+0000 7f750bfff700 1 -- 172.21.15.173:0/3306092283 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7514009320 con 0x7f75240f62a0 2024-01-21T19:53:07.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.717+0000 7f7528a4b700 1 -- 172.21.15.173:0/3306092283 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7524106ba0 con 0x7f75240f62a0 2024-01-21T19:53:07.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.717+0000 7f750bfff700 1 -- 172.21.15.173:0/3306092283 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7514013f00 con 0x7f75240f62a0 2024-01-21T19:53:07.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.717+0000 7f750bfff700 1 -- 172.21.15.173:0/3306092283 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7514014dc0 con 0x7f75240f62a0 2024-01-21T19:53:07.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.717+0000 7f7528a4b700 1 -- 172.21.15.173:0/3306092283 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7524107030 con 0x7f75240f62a0 2024-01-21T19:53:07.712 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.718+0000 7f750bfff700 1 -- 172.21.15.173:0/3306092283 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7514014420 con 0x7f75240f62a0 2024-01-21T19:53:07.712 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.718+0000 7f7528a4b700 1 -- 172.21.15.173:0/3306092283 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7524045ad0 con 0x7f75240f62a0 2024-01-21T19:53:07.712 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.718+0000 7f750bfff700 1 --2- 172.21.15.173:0/3306092283 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f750c02ff60 0x7f750c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:07.712 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.718+0000 7f750bfff700 1 -- 172.21.15.173:0/3306092283 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7514041020 con 0x7f75240f62a0 2024-01-21T19:53:07.713 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.718+0000 7f7522ffd700 1 --2- 172.21.15.173:0/3306092283 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f750c02ff60 0x7f750c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:07.713 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.718+0000 7f7522ffd700 1 --2- 172.21.15.173:0/3306092283 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f750c02ff60 0x7f750c032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f75180096a0 tx=0x7f7518006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:07.715 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.722+0000 7f750bfff700 1 -- 172.21.15.173:0/3306092283 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f75140095f0 con 0x7f75240f62a0 2024-01-21T19:53:07.919 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.926+0000 7f7528a4b700 1 -- 172.21.15.173:0/3306092283 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7524107e80 con 0x7f75240f62a0 2024-01-21T19:53:07.919 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.927+0000 7f750bfff700 1 -- 172.21.15.173:0/3306092283 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f751401cb10 con 0x7f75240f62a0 2024-01-21T19:53:07.920 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:53:07.920 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:53:07.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.928+0000 7f7528a4b700 1 -- 172.21.15.173:0/3306092283 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f750c02ff60 msgr2=0x7f750c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:07.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.928+0000 7f7528a4b700 1 --2- 172.21.15.173:0/3306092283 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f750c02ff60 0x7f750c032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f75180096a0 tx=0x7f7518006b40).stop 2024-01-21T19:53:07.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.928+0000 7f7528a4b700 1 -- 172.21.15.173:0/3306092283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75240f62a0 msgr2=0x7f75241064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:07.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.928+0000 7f7528a4b700 1 --2- 172.21.15.173:0/3306092283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75240f62a0 0x7f75241064b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f7514013510 tx=0x7f751400b040).stop 2024-01-21T19:53:07.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.928+0000 7f7528a4b700 1 -- 172.21.15.173:0/3306092283 shutdown_connections 2024-01-21T19:53:07.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.928+0000 7f7528a4b700 1 --2- 172.21.15.173:0/3306092283 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f750c02ff60 0x7f750c032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:07.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.928+0000 7f7528a4b700 1 --2- 172.21.15.173:0/3306092283 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f75240f62a0 0x7f75241064b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:07.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.928+0000 7f7528a4b700 1 -- 172.21.15.173:0/3306092283 >> 172.21.15.173:0/3306092283 conn(0x7f75240f1aa0 msgr2=0x7f75240f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:07.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.928+0000 7f7528a4b700 1 -- 172.21.15.173:0/3306092283 shutdown_connections 2024-01-21T19:53:07.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:07.928+0000 7f7528a4b700 1 -- 172.21.15.173:0/3306092283 wait complete. 2024-01-21T19:53:07.922 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:09.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:08 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:06.976041+0000 mgr.smithi169.espgue (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:09.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:08 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:07.928378+0000 mon.smithi169 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.173:0/3306092283' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:53:09.323 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:53:09.323 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:53:10.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.600+0000 7f10b1d06700 1 -- 172.21.15.173:0/3312595321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10ac0f5410 msgr2=0x7f10ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:10.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.600+0000 7f10b1d06700 1 --2- 172.21.15.173:0/3312595321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10ac0f5410 0x7f10ac0f5830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f1094009fb0 tx=0x7f1094009b30).stop 2024-01-21T19:53:10.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.600+0000 7f10b1d06700 1 -- 172.21.15.173:0/3312595321 shutdown_connections 2024-01-21T19:53:10.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.600+0000 7f10b1d06700 1 --2- 172.21.15.173:0/3312595321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10ac0f5410 0x7f10ac0f5830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:10.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.600+0000 7f10b1d06700 1 -- 172.21.15.173:0/3312595321 >> 172.21.15.173:0/3312595321 conn(0x7f10ac0f0b70 msgr2=0x7f10ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:10.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.600+0000 7f10b1d06700 1 -- 172.21.15.173:0/3312595321 shutdown_connections 2024-01-21T19:53:10.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.600+0000 7f10b1d06700 1 -- 172.21.15.173:0/3312595321 wait complete. 2024-01-21T19:53:10.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.601+0000 7f10b1d06700 1 Processor -- start 2024-01-21T19:53:10.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.601+0000 7f10b1d06700 1 -- start start 2024-01-21T19:53:10.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.601+0000 7f10b1d06700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10ac0f5410 0x7f10ac106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:10.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.601+0000 7f10b1d06700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ac106c50 con 0x7f10ac0f5410 2024-01-21T19:53:10.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.602+0000 7f10ab7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10ac0f5410 0x7f10ac106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:10.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.602+0000 7f10ab7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10ac0f5410 0x7f10ac106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:34584/0 (socket says 172.21.15.173:34584) 2024-01-21T19:53:10.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.602+0000 7f10ab7fe700 1 -- 172.21.15.173:0/579630095 learned_addr learned my addr 172.21.15.173:0/579630095 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:53:10.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.602+0000 7f10ab7fe700 1 -- 172.21.15.173:0/579630095 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1094005040 con 0x7f10ac0f5410 2024-01-21T19:53:10.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.602+0000 7f10ab7fe700 1 --2- 172.21.15.173:0/579630095 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10ac0f5410 0x7f10ac106760 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f1094009f80 tx=0x7f109400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:10.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.602+0000 7f10a3fff700 1 -- 172.21.15.173:0/579630095 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1094009320 con 0x7f10ac0f5410 2024-01-21T19:53:10.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.602+0000 7f10a3fff700 1 -- 172.21.15.173:0/579630095 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1094013ec0 con 0x7f10ac0f5410 2024-01-21T19:53:10.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.602+0000 7f10b1d06700 1 -- 172.21.15.173:0/579630095 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10ac106e50 con 0x7f10ac0f5410 2024-01-21T19:53:10.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.602+0000 7f10a3fff700 1 -- 172.21.15.173:0/579630095 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f109401ece0 con 0x7f10ac0f5410 2024-01-21T19:53:10.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.602+0000 7f10b1d06700 1 -- 172.21.15.173:0/579630095 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10ac1072e0 con 0x7f10ac0f5410 2024-01-21T19:53:10.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.603+0000 7f10a3fff700 1 -- 172.21.15.173:0/579630095 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f109401e420 con 0x7f10ac0f5410 2024-01-21T19:53:10.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.603+0000 7f10b1d06700 1 -- 172.21.15.173:0/579630095 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10ac045ad0 con 0x7f10ac0f5410 2024-01-21T19:53:10.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.603+0000 7f10a3fff700 1 --2- 172.21.15.173:0/579630095 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f109802ff60 0x7f1098032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:10.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.603+0000 7f10a3fff700 1 -- 172.21.15.173:0/579630095 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1094040ea0 con 0x7f10ac0f5410 2024-01-21T19:53:10.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.603+0000 7f10aaffd700 1 --2- 172.21.15.173:0/579630095 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f109802ff60 0x7f1098032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:10.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.604+0000 7f10aaffd700 1 --2- 172.21.15.173:0/579630095 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f109802ff60 0x7f1098032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f109c0096a0 tx=0x7f109c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:10.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.608+0000 7f10a3fff700 1 -- 172.21.15.173:0/579630095 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f109401e730 con 0x7f10ac0f5410 2024-01-21T19:53:10.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.813+0000 7f10b1d06700 1 -- 172.21.15.173:0/579630095 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f10ac108190 con 0x7f10ac0f5410 2024-01-21T19:53:10.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.814+0000 7f10a3fff700 1 -- 172.21.15.173:0/579630095 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f109402c2d0 con 0x7f10ac0f5410 2024-01-21T19:53:10.806 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:53:10.806 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:53:10.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.815+0000 7f10b1d06700 1 -- 172.21.15.173:0/579630095 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f109802ff60 msgr2=0x7f1098032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:10.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.815+0000 7f10b1d06700 1 --2- 172.21.15.173:0/579630095 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f109802ff60 0x7f1098032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f109c0096a0 tx=0x7f109c006b40).stop 2024-01-21T19:53:10.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.815+0000 7f10b1d06700 1 -- 172.21.15.173:0/579630095 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10ac0f5410 msgr2=0x7f10ac106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:10.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.815+0000 7f10b1d06700 1 --2- 172.21.15.173:0/579630095 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10ac0f5410 0x7f10ac106760 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f1094009f80 tx=0x7f109400b040).stop 2024-01-21T19:53:10.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.815+0000 7f10b1d06700 1 -- 172.21.15.173:0/579630095 shutdown_connections 2024-01-21T19:53:10.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.815+0000 7f10b1d06700 1 --2- 172.21.15.173:0/579630095 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f109802ff60 0x7f1098032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:10.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.815+0000 7f10b1d06700 1 --2- 172.21.15.173:0/579630095 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10ac0f5410 0x7f10ac106760 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:10.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.816+0000 7f10b1d06700 1 -- 172.21.15.173:0/579630095 >> 172.21.15.173:0/579630095 conn(0x7f10ac0f0b70 msgr2=0x7f10ac05fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:10.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.816+0000 7f10b1d06700 1 -- 172.21.15.173:0/579630095 shutdown_connections 2024-01-21T19:53:10.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:10.816+0000 7f10b1d06700 1 -- 172.21.15.173:0/579630095 wait complete. 2024-01-21T19:53:10.808 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:11.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:10 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:08.976313+0000 mgr.smithi169.espgue (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:12.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:11 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:10.814650+0000 mon.smithi169 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.173:0/579630095' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:53:12.211 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:53:12.212 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:53:13.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:12 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:10.976570+0000 mgr.smithi169.espgue (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:13.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.434+0000 7f072e1d7700 1 -- 172.21.15.173:0/226948661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0728060280 msgr2=0x7f07280f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:13.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.434+0000 7f072e1d7700 1 --2- 172.21.15.173:0/226948661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0728060280 0x7f07280f90f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f0718009fb0 tx=0x7f0718009b30).stop 2024-01-21T19:53:13.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.434+0000 7f072e1d7700 1 -- 172.21.15.173:0/226948661 shutdown_connections 2024-01-21T19:53:13.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.434+0000 7f072e1d7700 1 --2- 172.21.15.173:0/226948661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0728060280 0x7f07280f90f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:13.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.434+0000 7f072e1d7700 1 -- 172.21.15.173:0/226948661 >> 172.21.15.173:0/226948661 conn(0x7f07280f0b70 msgr2=0x7f07280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:13.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.434+0000 7f072e1d7700 1 -- 172.21.15.173:0/226948661 shutdown_connections 2024-01-21T19:53:13.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.434+0000 7f072e1d7700 1 -- 172.21.15.173:0/226948661 wait complete. 2024-01-21T19:53:13.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.435+0000 7f072e1d7700 1 Processor -- start 2024-01-21T19:53:13.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.435+0000 7f072e1d7700 1 -- start start 2024-01-21T19:53:13.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.435+0000 7f072e1d7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0728060280 0x7f07281066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:13.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.435+0000 7f072e1d7700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0728106bb0 con 0x7f0728060280 2024-01-21T19:53:13.429 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.435+0000 7f07277fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0728060280 0x7f07281066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:13.429 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.435+0000 7f07277fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0728060280 0x7f07281066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:47294/0 (socket says 172.21.15.173:47294) 2024-01-21T19:53:13.429 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.435+0000 7f07277fe700 1 -- 172.21.15.173:0/3828011024 learned_addr learned my addr 172.21.15.173:0/3828011024 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:53:13.429 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.436+0000 7f07277fe700 1 -- 172.21.15.173:0/3828011024 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0718005040 con 0x7f0728060280 2024-01-21T19:53:13.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.436+0000 7f07277fe700 1 --2- 172.21.15.173:0/3828011024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0728060280 0x7f07281066c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f0718013510 tx=0x7f071800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:13.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.436+0000 7f071ffff700 1 -- 172.21.15.173:0/3828011024 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0718009320 con 0x7f0728060280 2024-01-21T19:53:13.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.436+0000 7f071ffff700 1 -- 172.21.15.173:0/3828011024 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0718013f00 con 0x7f0728060280 2024-01-21T19:53:13.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.436+0000 7f072e1d7700 1 -- 172.21.15.173:0/3828011024 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0728106db0 con 0x7f0728060280 2024-01-21T19:53:13.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.436+0000 7f071ffff700 1 -- 172.21.15.173:0/3828011024 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0718014dc0 con 0x7f0728060280 2024-01-21T19:53:13.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.436+0000 7f072e1d7700 1 -- 172.21.15.173:0/3828011024 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0728107240 con 0x7f0728060280 2024-01-21T19:53:13.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.437+0000 7f071ffff700 1 -- 172.21.15.173:0/3828011024 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0718014420 con 0x7f0728060280 2024-01-21T19:53:13.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.437+0000 7f072e1d7700 1 -- 172.21.15.173:0/3828011024 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0728045ad0 con 0x7f0728060280 2024-01-21T19:53:13.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.437+0000 7f071ffff700 1 --2- 172.21.15.173:0/3828011024 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f071402ff60 0x7f0714032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:13.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.437+0000 7f071ffff700 1 -- 172.21.15.173:0/3828011024 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f07180410f0 con 0x7f0728060280 2024-01-21T19:53:13.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.437+0000 7f0726ffd700 1 --2- 172.21.15.173:0/3828011024 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f071402ff60 0x7f0714032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:13.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.438+0000 7f0726ffd700 1 --2- 172.21.15.173:0/3828011024 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f071402ff60 0x7f0714032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f07100096a0 tx=0x7f0710006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:13.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.442+0000 7f071ffff700 1 -- 172.21.15.173:0/3828011024 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07180095f0 con 0x7f0728060280 2024-01-21T19:53:13.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.651+0000 7f072e1d7700 1 -- 172.21.15.173:0/3828011024 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07281080f0 con 0x7f0728060280 2024-01-21T19:53:13.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.651+0000 7f071ffff700 1 -- 172.21.15.173:0/3828011024 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f071801cb10 con 0x7f0728060280 2024-01-21T19:53:13.643 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:53:13.644 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:53:13.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.653+0000 7f072e1d7700 1 -- 172.21.15.173:0/3828011024 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f071402ff60 msgr2=0x7f0714032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:13.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.653+0000 7f072e1d7700 1 --2- 172.21.15.173:0/3828011024 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f071402ff60 0x7f0714032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f07100096a0 tx=0x7f0710006b40).stop 2024-01-21T19:53:13.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.653+0000 7f072e1d7700 1 -- 172.21.15.173:0/3828011024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0728060280 msgr2=0x7f07281066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:13.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.653+0000 7f072e1d7700 1 --2- 172.21.15.173:0/3828011024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0728060280 0x7f07281066c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f0718013510 tx=0x7f071800b040).stop 2024-01-21T19:53:13.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.653+0000 7f072e1d7700 1 -- 172.21.15.173:0/3828011024 shutdown_connections 2024-01-21T19:53:13.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.653+0000 7f072e1d7700 1 --2- 172.21.15.173:0/3828011024 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f071402ff60 0x7f0714032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:13.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.653+0000 7f072e1d7700 1 --2- 172.21.15.173:0/3828011024 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0728060280 0x7f07281066c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:13.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.653+0000 7f072e1d7700 1 -- 172.21.15.173:0/3828011024 >> 172.21.15.173:0/3828011024 conn(0x7f07280f0b70 msgr2=0x7f07280f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:13.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.653+0000 7f072e1d7700 1 -- 172.21.15.173:0/3828011024 shutdown_connections 2024-01-21T19:53:13.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:13.653+0000 7f072e1d7700 1 -- 172.21.15.173:0/3828011024 wait complete. 2024-01-21T19:53:13.646 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:14.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:13 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:13.652182+0000 mon.smithi169 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.173:0/3828011024' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:53:15.077 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:53:15.077 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:53:15.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:14 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:12.976901+0000 mgr.smithi169.espgue (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:16.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.274+0000 7f13ca337700 1 -- 172.21.15.173:0/850879481 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13c40f5410 msgr2=0x7f13c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:16.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.274+0000 7f13ca337700 1 --2- 172.21.15.173:0/850879481 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13c40f5410 0x7f13c40f5830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f13b4009fb0 tx=0x7f13b4009b30).stop 2024-01-21T19:53:16.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.274+0000 7f13ca337700 1 -- 172.21.15.173:0/850879481 shutdown_connections 2024-01-21T19:53:16.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.274+0000 7f13ca337700 1 --2- 172.21.15.173:0/850879481 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13c40f5410 0x7f13c40f5830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:16.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.274+0000 7f13ca337700 1 -- 172.21.15.173:0/850879481 >> 172.21.15.173:0/850879481 conn(0x7f13c40f0b70 msgr2=0x7f13c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:16.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.274+0000 7f13ca337700 1 -- 172.21.15.173:0/850879481 shutdown_connections 2024-01-21T19:53:16.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.274+0000 7f13ca337700 1 -- 172.21.15.173:0/850879481 wait complete. 2024-01-21T19:53:16.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.274+0000 7f13ca337700 1 Processor -- start 2024-01-21T19:53:16.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.274+0000 7f13ca337700 1 -- start start 2024-01-21T19:53:16.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.275+0000 7f13ca337700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13c40f5410 0x7f13c40fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:16.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.275+0000 7f13ca337700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c40fb820 con 0x7f13c40f5410 2024-01-21T19:53:16.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.275+0000 7f13c3fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13c40f5410 0x7f13c40fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:16.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.275+0000 7f13c3fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13c40f5410 0x7f13c40fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:47316/0 (socket says 172.21.15.173:47316) 2024-01-21T19:53:16.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.275+0000 7f13c3fff700 1 -- 172.21.15.173:0/1839353509 learned_addr learned my addr 172.21.15.173:0/1839353509 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:53:16.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.275+0000 7f13c3fff700 1 -- 172.21.15.173:0/1839353509 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b4005040 con 0x7f13c40f5410 2024-01-21T19:53:16.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.275+0000 7f13c3fff700 1 --2- 172.21.15.173:0/1839353509 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13c40f5410 0x7f13c40fd3a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f13b4014000 tx=0x7f13b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:16.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.275+0000 7f13c0ff9700 1 -- 172.21.15.173:0/1839353509 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13b40095f0 con 0x7f13c40f5410 2024-01-21T19:53:16.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.276+0000 7f13ca337700 1 -- 172.21.15.173:0/1839353509 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c40fb9c0 con 0x7f13c40f5410 2024-01-21T19:53:16.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.276+0000 7f13c0ff9700 1 -- 172.21.15.173:0/1839353509 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f13b4007e70 con 0x7f13c40f5410 2024-01-21T19:53:16.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.276+0000 7f13c0ff9700 1 -- 172.21.15.173:0/1839353509 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13b4014ea0 con 0x7f13c40f5410 2024-01-21T19:53:16.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.276+0000 7f13ca337700 1 -- 172.21.15.173:0/1839353509 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c40fbe50 con 0x7f13c40f5410 2024-01-21T19:53:16.269 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.276+0000 7f13c0ff9700 1 -- 172.21.15.173:0/1839353509 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13b4025070 con 0x7f13c40f5410 2024-01-21T19:53:16.269 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.276+0000 7f13ca337700 1 -- 172.21.15.173:0/1839353509 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c4059100 con 0x7f13c40f5410 2024-01-21T19:53:16.270 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.276+0000 7f13c0ff9700 1 --2- 172.21.15.173:0/1839353509 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f13ac02ff60 0x7f13ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:16.270 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.277+0000 7f13c0ff9700 1 -- 172.21.15.173:0/1839353509 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f13b4040930 con 0x7f13c40f5410 2024-01-21T19:53:16.270 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.277+0000 7f13c37fe700 1 --2- 172.21.15.173:0/1839353509 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f13ac02ff60 0x7f13ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:16.270 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.277+0000 7f13c37fe700 1 --2- 172.21.15.173:0/1839353509 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f13ac02ff60 0x7f13ac032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f13b80096a0 tx=0x7f13b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:16.273 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.281+0000 7f13c0ff9700 1 -- 172.21.15.173:0/1839353509 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f13b400f4e0 con 0x7f13c40f5410 2024-01-21T19:53:16.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.487+0000 7f13ca337700 1 -- 172.21.15.173:0/1839353509 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13c40fae60 con 0x7f13c40f5410 2024-01-21T19:53:16.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.488+0000 7f13c0ff9700 1 -- 172.21.15.173:0/1839353509 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f13b400f300 con 0x7f13c40f5410 2024-01-21T19:53:16.480 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:53:16.480 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:53:16.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.489+0000 7f13ca337700 1 -- 172.21.15.173:0/1839353509 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f13ac02ff60 msgr2=0x7f13ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:16.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.489+0000 7f13ca337700 1 --2- 172.21.15.173:0/1839353509 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f13ac02ff60 0x7f13ac032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f13b80096a0 tx=0x7f13b8006b40).stop 2024-01-21T19:53:16.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.489+0000 7f13ca337700 1 -- 172.21.15.173:0/1839353509 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13c40f5410 msgr2=0x7f13c40fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:16.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.489+0000 7f13ca337700 1 --2- 172.21.15.173:0/1839353509 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13c40f5410 0x7f13c40fd3a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f13b4014000 tx=0x7f13b400b040).stop 2024-01-21T19:53:16.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.489+0000 7f13ca337700 1 -- 172.21.15.173:0/1839353509 shutdown_connections 2024-01-21T19:53:16.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.489+0000 7f13ca337700 1 --2- 172.21.15.173:0/1839353509 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f13ac02ff60 0x7f13ac032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:16.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.489+0000 7f13ca337700 1 --2- 172.21.15.173:0/1839353509 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13c40f5410 0x7f13c40fd3a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:16.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.489+0000 7f13ca337700 1 -- 172.21.15.173:0/1839353509 >> 172.21.15.173:0/1839353509 conn(0x7f13c40f0b70 msgr2=0x7f13c405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:16.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.489+0000 7f13ca337700 1 -- 172.21.15.173:0/1839353509 shutdown_connections 2024-01-21T19:53:16.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:16.489+0000 7f13ca337700 1 -- 172.21.15.173:0/1839353509 wait complete. 2024-01-21T19:53:16.482 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:17.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:16 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:14.977255+0000 mgr.smithi169.espgue (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:17.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:16 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:16.488544+0000 mon.smithi169 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.173:0/1839353509' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:53:17.882 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:53:17.882 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:53:19.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.077+0000 7fbd4c2bd700 1 -- 172.21.15.173:0/3233840966 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd440f6510 msgr2=0x7fbd440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:19.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.077+0000 7fbd4c2bd700 1 --2- 172.21.15.173:0/3233840966 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd440f6510 0x7fbd440f6930 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fbd34009fb0 tx=0x7fbd34009b30).stop 2024-01-21T19:53:19.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.077+0000 7fbd4c2bd700 1 -- 172.21.15.173:0/3233840966 shutdown_connections 2024-01-21T19:53:19.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.077+0000 7fbd4c2bd700 1 --2- 172.21.15.173:0/3233840966 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd440f6510 0x7fbd440f6930 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:19.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.077+0000 7fbd4c2bd700 1 -- 172.21.15.173:0/3233840966 >> 172.21.15.173:0/3233840966 conn(0x7fbd440f1c70 msgr2=0x7fbd440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:19.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.077+0000 7fbd4c2bd700 1 -- 172.21.15.173:0/3233840966 shutdown_connections 2024-01-21T19:53:19.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.077+0000 7fbd4c2bd700 1 -- 172.21.15.173:0/3233840966 wait complete. 2024-01-21T19:53:19.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.078+0000 7fbd4c2bd700 1 Processor -- start 2024-01-21T19:53:19.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.078+0000 7fbd4c2bd700 1 -- start start 2024-01-21T19:53:19.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.078+0000 7fbd4c2bd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd440f6510 0x7fbd4406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:19.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.078+0000 7fbd4c2bd700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd44069e20 con 0x7fbd440f6510 2024-01-21T19:53:19.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.078+0000 7fbd4a059700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd440f6510 0x7fbd4406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:19.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.078+0000 7fbd4a059700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd440f6510 0x7fbd4406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:47336/0 (socket says 172.21.15.173:47336) 2024-01-21T19:53:19.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.078+0000 7fbd4a059700 1 -- 172.21.15.173:0/4192408505 learned_addr learned my addr 172.21.15.173:0/4192408505 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:53:19.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.078+0000 7fbd4a059700 1 -- 172.21.15.173:0/4192408505 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd34005040 con 0x7fbd440f6510 2024-01-21T19:53:19.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.078+0000 7fbd4a059700 1 --2- 172.21.15.173:0/4192408505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd440f6510 0x7fbd4406b9a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fbd34013510 tx=0x7fbd3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:19.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.078+0000 7fbd3affd700 1 -- 172.21.15.173:0/4192408505 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd34009320 con 0x7fbd440f6510 2024-01-21T19:53:19.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.079+0000 7fbd3affd700 1 -- 172.21.15.173:0/4192408505 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd34013f00 con 0x7fbd440f6510 2024-01-21T19:53:19.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.079+0000 7fbd4c2bd700 1 -- 172.21.15.173:0/4192408505 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd44069fc0 con 0x7fbd440f6510 2024-01-21T19:53:19.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.079+0000 7fbd3affd700 1 -- 172.21.15.173:0/4192408505 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd34014d90 con 0x7fbd440f6510 2024-01-21T19:53:19.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.079+0000 7fbd4c2bd700 1 -- 172.21.15.173:0/4192408505 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd4406a450 con 0x7fbd440f6510 2024-01-21T19:53:19.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.079+0000 7fbd3affd700 1 -- 172.21.15.173:0/4192408505 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd340095f0 con 0x7fbd440f6510 2024-01-21T19:53:19.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.079+0000 7fbd4c2bd700 1 -- 172.21.15.173:0/4192408505 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd44059100 con 0x7fbd440f6510 2024-01-21T19:53:19.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.080+0000 7fbd3affd700 1 --2- 172.21.15.173:0/4192408505 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fbd3002ff60 0x7fbd30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:19.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.080+0000 7fbd3affd700 1 -- 172.21.15.173:0/4192408505 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd34040fc0 con 0x7fbd440f6510 2024-01-21T19:53:19.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.080+0000 7fbd49858700 1 --2- 172.21.15.173:0/4192408505 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fbd3002ff60 0x7fbd30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:19.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.080+0000 7fbd49858700 1 --2- 172.21.15.173:0/4192408505 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fbd3002ff60 0x7fbd30032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fbd400096a0 tx=0x7fbd40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:19.076 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.084+0000 7fbd3affd700 1 -- 172.21.15.173:0/4192408505 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd34014420 con 0x7fbd440f6510 2024-01-21T19:53:19.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:18 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:16.977582+0000 mgr.smithi169.espgue (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:19.280 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.288+0000 7fbd4c2bd700 1 -- 172.21.15.173:0/4192408505 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd44045ad0 con 0x7fbd440f6510 2024-01-21T19:53:19.281 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.289+0000 7fbd3affd700 1 -- 172.21.15.173:0/4192408505 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbd34010230 con 0x7fbd440f6510 2024-01-21T19:53:19.281 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:53:19.281 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:53:19.282 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.291+0000 7fbd4c2bd700 1 -- 172.21.15.173:0/4192408505 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fbd3002ff60 msgr2=0x7fbd30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:19.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.291+0000 7fbd4c2bd700 1 --2- 172.21.15.173:0/4192408505 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fbd3002ff60 0x7fbd30032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fbd400096a0 tx=0x7fbd40006b40).stop 2024-01-21T19:53:19.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.291+0000 7fbd4c2bd700 1 -- 172.21.15.173:0/4192408505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd440f6510 msgr2=0x7fbd4406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:19.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.291+0000 7fbd4c2bd700 1 --2- 172.21.15.173:0/4192408505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd440f6510 0x7fbd4406b9a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fbd34013510 tx=0x7fbd3400b040).stop 2024-01-21T19:53:19.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.291+0000 7fbd4c2bd700 1 -- 172.21.15.173:0/4192408505 shutdown_connections 2024-01-21T19:53:19.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.291+0000 7fbd4c2bd700 1 --2- 172.21.15.173:0/4192408505 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fbd3002ff60 0x7fbd30032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:19.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.291+0000 7fbd4c2bd700 1 --2- 172.21.15.173:0/4192408505 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbd440f6510 0x7fbd4406b9a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:19.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.291+0000 7fbd4c2bd700 1 -- 172.21.15.173:0/4192408505 >> 172.21.15.173:0/4192408505 conn(0x7fbd440f1c70 msgr2=0x7fbd440f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:19.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.291+0000 7fbd4c2bd700 1 -- 172.21.15.173:0/4192408505 shutdown_connections 2024-01-21T19:53:19.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:19.291+0000 7fbd4c2bd700 1 -- 172.21.15.173:0/4192408505 wait complete. 2024-01-21T19:53:19.283 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:20.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:19 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:19.289846+0000 mon.smithi169 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.173:0/4192408505' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:53:20.690 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:53:20.690 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:53:21.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:20 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:18.977889+0000 mgr.smithi169.espgue (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:21.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:21.999+0000 7f0250905700 1 -- 172.21.15.173:0/2571891209 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f024c0f5410 msgr2=0x7f024c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:21.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:21.999+0000 7f0250905700 1 --2- 172.21.15.173:0/2571891209 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f024c0f5410 0x7f024c0f5830 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f0240009fb0 tx=0x7f0240009b30).stop 2024-01-21T19:53:21.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:21.999+0000 7f0250905700 1 -- 172.21.15.173:0/2571891209 shutdown_connections 2024-01-21T19:53:21.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:21.999+0000 7f0250905700 1 --2- 172.21.15.173:0/2571891209 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f024c0f5410 0x7f024c0f5830 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:21.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.000+0000 7f0250905700 1 -- 172.21.15.173:0/2571891209 >> 172.21.15.173:0/2571891209 conn(0x7f024c0f0b70 msgr2=0x7f024c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:21.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.000+0000 7f0250905700 1 -- 172.21.15.173:0/2571891209 shutdown_connections 2024-01-21T19:53:21.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.000+0000 7f0250905700 1 -- 172.21.15.173:0/2571891209 wait complete. 2024-01-21T19:53:21.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.000+0000 7f0250905700 1 Processor -- start 2024-01-21T19:53:21.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.000+0000 7f0250905700 1 -- start start 2024-01-21T19:53:21.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.000+0000 7f0250905700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f024c0f5410 0x7f024c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:21.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.000+0000 7f0250905700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024c105b20 con 0x7f024c0f5410 2024-01-21T19:53:21.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.001+0000 7f024a59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f024c0f5410 0x7f024c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:21.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.001+0000 7f024a59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f024c0f5410 0x7f024c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:47356/0 (socket says 172.21.15.173:47356) 2024-01-21T19:53:21.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.001+0000 7f024a59c700 1 -- 172.21.15.173:0/773529152 learned_addr learned my addr 172.21.15.173:0/773529152 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:53:21.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.001+0000 7f024a59c700 1 -- 172.21.15.173:0/773529152 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0240005040 con 0x7f024c0f5410 2024-01-21T19:53:21.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.001+0000 7f024a59c700 1 --2- 172.21.15.173:0/773529152 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f024c0f5410 0x7f024c105630 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f0240013510 tx=0x7f024000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:21.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.001+0000 7f023affd700 1 -- 172.21.15.173:0/773529152 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0240009320 con 0x7f024c0f5410 2024-01-21T19:53:21.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.001+0000 7f0250905700 1 -- 172.21.15.173:0/773529152 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f024c105d20 con 0x7f024c0f5410 2024-01-21T19:53:21.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.001+0000 7f023affd700 1 -- 172.21.15.173:0/773529152 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0240013f00 con 0x7f024c0f5410 2024-01-21T19:53:21.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.001+0000 7f023affd700 1 -- 172.21.15.173:0/773529152 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0240014d90 con 0x7f024c0f5410 2024-01-21T19:53:21.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.001+0000 7f0250905700 1 -- 172.21.15.173:0/773529152 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f024c1061b0 con 0x7f024c0f5410 2024-01-21T19:53:21.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.002+0000 7f023affd700 1 -- 172.21.15.173:0/773529152 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02400095f0 con 0x7f024c0f5410 2024-01-21T19:53:21.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.002+0000 7f0250905700 1 -- 172.21.15.173:0/773529152 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f024c045ad0 con 0x7f024c0f5410 2024-01-21T19:53:21.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.002+0000 7f023affd700 1 --2- 172.21.15.173:0/773529152 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f023402ff60 0x7f0234032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:21.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.002+0000 7f023affd700 1 -- 172.21.15.173:0/773529152 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0240041000 con 0x7f024c0f5410 2024-01-21T19:53:21.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.003+0000 7f0249d9b700 1 --2- 172.21.15.173:0/773529152 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f023402ff60 0x7f0234032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:21.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.003+0000 7f0249d9b700 1 --2- 172.21.15.173:0/773529152 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f023402ff60 0x7f0234032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f023c0096a0 tx=0x7f023c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:21.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.007+0000 7f023affd700 1 -- 172.21.15.173:0/773529152 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0240014420 con 0x7f024c0f5410 2024-01-21T19:53:22.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.213+0000 7f0250905700 1 -- 172.21.15.173:0/773529152 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f024c0fae60 con 0x7f024c0f5410 2024-01-21T19:53:22.206 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.214+0000 7f023affd700 1 -- 172.21.15.173:0/773529152 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0240010230 con 0x7f024c0f5410 2024-01-21T19:53:22.206 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:53:22.206 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:53:22.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.216+0000 7f0250905700 1 -- 172.21.15.173:0/773529152 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f023402ff60 msgr2=0x7f0234032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:22.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.216+0000 7f0250905700 1 --2- 172.21.15.173:0/773529152 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f023402ff60 0x7f0234032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f023c0096a0 tx=0x7f023c006b40).stop 2024-01-21T19:53:22.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.216+0000 7f0250905700 1 -- 172.21.15.173:0/773529152 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f024c0f5410 msgr2=0x7f024c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:22.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.216+0000 7f0250905700 1 --2- 172.21.15.173:0/773529152 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f024c0f5410 0x7f024c105630 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f0240013510 tx=0x7f024000b040).stop 2024-01-21T19:53:22.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.216+0000 7f0250905700 1 -- 172.21.15.173:0/773529152 shutdown_connections 2024-01-21T19:53:22.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.216+0000 7f0250905700 1 --2- 172.21.15.173:0/773529152 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f023402ff60 0x7f0234032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:22.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.216+0000 7f0250905700 1 --2- 172.21.15.173:0/773529152 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f024c0f5410 0x7f024c105630 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:22.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.216+0000 7f0250905700 1 -- 172.21.15.173:0/773529152 >> 172.21.15.173:0/773529152 conn(0x7f024c0f0b70 msgr2=0x7f024c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:22.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.216+0000 7f0250905700 1 -- 172.21.15.173:0/773529152 shutdown_connections 2024-01-21T19:53:22.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:22.216+0000 7f0250905700 1 -- 172.21.15.173:0/773529152 wait complete. 2024-01-21T19:53:22.208 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:23.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:22 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:20.978188+0000 mgr.smithi169.espgue (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:23.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:22 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:22.214731+0000 mon.smithi169 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.173:0/773529152' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:53:23.602 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:53:23.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:53:24.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.815+0000 7fa058a73700 1 -- 172.21.15.173:0/1830634693 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0540f6510 msgr2=0x7fa0540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:24.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.815+0000 7fa058a73700 1 --2- 172.21.15.173:0/1830634693 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0540f6510 0x7fa0540f6930 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fa044009fb0 tx=0x7fa044009b30).stop 2024-01-21T19:53:24.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.815+0000 7fa058a73700 1 -- 172.21.15.173:0/1830634693 shutdown_connections 2024-01-21T19:53:24.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.815+0000 7fa058a73700 1 --2- 172.21.15.173:0/1830634693 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0540f6510 0x7fa0540f6930 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:24.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.815+0000 7fa058a73700 1 -- 172.21.15.173:0/1830634693 >> 172.21.15.173:0/1830634693 conn(0x7fa0540f1c70 msgr2=0x7fa0540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:24.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.815+0000 7fa058a73700 1 -- 172.21.15.173:0/1830634693 shutdown_connections 2024-01-21T19:53:24.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.815+0000 7fa058a73700 1 -- 172.21.15.173:0/1830634693 wait complete. 2024-01-21T19:53:24.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.816+0000 7fa058a73700 1 Processor -- start 2024-01-21T19:53:24.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.816+0000 7fa058a73700 1 -- start start 2024-01-21T19:53:24.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.816+0000 7fa058a73700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0540f6510 0x7fa054106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:24.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.816+0000 7fa058a73700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa054106c20 con 0x7fa0540f6510 2024-01-21T19:53:24.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.817+0000 7fa05259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0540f6510 0x7fa054106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:24.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.817+0000 7fa05259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0540f6510 0x7fa054106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:33598/0 (socket says 172.21.15.173:33598) 2024-01-21T19:53:24.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.817+0000 7fa05259c700 1 -- 172.21.15.173:0/1008495945 learned_addr learned my addr 172.21.15.173:0/1008495945 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:53:24.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.817+0000 7fa05259c700 1 -- 172.21.15.173:0/1008495945 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa044005040 con 0x7fa0540f6510 2024-01-21T19:53:24.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.817+0000 7fa05259c700 1 --2- 172.21.15.173:0/1008495945 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0540f6510 0x7fa054106730 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fa044014000 tx=0x7fa04400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:24.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.817+0000 7fa042ffd700 1 -- 172.21.15.173:0/1008495945 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0440095f0 con 0x7fa0540f6510 2024-01-21T19:53:24.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.817+0000 7fa042ffd700 1 -- 172.21.15.173:0/1008495945 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa044007e70 con 0x7fa0540f6510 2024-01-21T19:53:24.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.817+0000 7fa058a73700 1 -- 172.21.15.173:0/1008495945 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa054106e20 con 0x7fa0540f6510 2024-01-21T19:53:24.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.817+0000 7fa042ffd700 1 -- 172.21.15.173:0/1008495945 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa044014ea0 con 0x7fa0540f6510 2024-01-21T19:53:24.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.817+0000 7fa058a73700 1 -- 172.21.15.173:0/1008495945 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0541072b0 con 0x7fa0540f6510 2024-01-21T19:53:24.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.818+0000 7fa042ffd700 1 -- 172.21.15.173:0/1008495945 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa044025070 con 0x7fa0540f6510 2024-01-21T19:53:24.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.818+0000 7fa042ffd700 1 --2- 172.21.15.173:0/1008495945 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa03c02ff60 0x7fa03c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:24.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.818+0000 7fa058a73700 1 -- 172.21.15.173:0/1008495945 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa054045ad0 con 0x7fa0540f6510 2024-01-21T19:53:24.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.818+0000 7fa042ffd700 1 -- 172.21.15.173:0/1008495945 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa0440409e0 con 0x7fa0540f6510 2024-01-21T19:53:24.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.818+0000 7fa051d9b700 1 --2- 172.21.15.173:0/1008495945 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa03c02ff60 0x7fa03c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:24.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.819+0000 7fa051d9b700 1 --2- 172.21.15.173:0/1008495945 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa03c02ff60 0x7fa03c032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fa0480096a0 tx=0x7fa048006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:24.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:24.823+0000 7fa042ffd700 1 -- 172.21.15.173:0/1008495945 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa0440079c0 con 0x7fa0540f6510 2024-01-21T19:53:25.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:25.027+0000 7fa058a73700 1 -- 172.21.15.173:0/1008495945 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0540fc050 con 0x7fa0540f6510 2024-01-21T19:53:25.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:25.027+0000 7fa042ffd700 1 -- 172.21.15.173:0/1008495945 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa04400f300 con 0x7fa0540f6510 2024-01-21T19:53:25.019 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:53:25.019 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:53:25.020 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:25.029+0000 7fa058a73700 1 -- 172.21.15.173:0/1008495945 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa03c02ff60 msgr2=0x7fa03c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:25.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:25.029+0000 7fa058a73700 1 --2- 172.21.15.173:0/1008495945 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa03c02ff60 0x7fa03c032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fa0480096a0 tx=0x7fa048006b40).stop 2024-01-21T19:53:25.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:25.029+0000 7fa058a73700 1 -- 172.21.15.173:0/1008495945 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0540f6510 msgr2=0x7fa054106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:25.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:25.029+0000 7fa058a73700 1 --2- 172.21.15.173:0/1008495945 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0540f6510 0x7fa054106730 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fa044014000 tx=0x7fa04400b040).stop 2024-01-21T19:53:25.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:25.029+0000 7fa058a73700 1 -- 172.21.15.173:0/1008495945 shutdown_connections 2024-01-21T19:53:25.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:25.029+0000 7fa058a73700 1 --2- 172.21.15.173:0/1008495945 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa03c02ff60 0x7fa03c032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:25.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:25.029+0000 7fa058a73700 1 --2- 172.21.15.173:0/1008495945 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa0540f6510 0x7fa054106730 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:25.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:25.029+0000 7fa058a73700 1 -- 172.21.15.173:0/1008495945 >> 172.21.15.173:0/1008495945 conn(0x7fa0540f1c70 msgr2=0x7fa0540f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:25.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:25.029+0000 7fa058a73700 1 -- 172.21.15.173:0/1008495945 shutdown_connections 2024-01-21T19:53:25.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:25.029+0000 7fa058a73700 1 -- 172.21.15.173:0/1008495945 wait complete. 2024-01-21T19:53:25.021 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:25.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:24 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:22.978494+0000 mgr.smithi169.espgue (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:26.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:25 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:25.028034+0000 mon.smithi169 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.173:0/1008495945' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:53:26.413 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:53:26.413 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:53:27.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:26 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:24.978764+0000 mgr.smithi169.espgue (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:27.579 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.586+0000 7f7d90429700 1 -- 172.21.15.173:0/101180383 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d880f6510 msgr2=0x7f7d880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:27.579 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.586+0000 7f7d90429700 1 --2- 172.21.15.173:0/101180383 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d880f6510 0x7f7d880f6930 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f7d78009fb0 tx=0x7f7d78009b30).stop 2024-01-21T19:53:27.579 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.586+0000 7f7d90429700 1 -- 172.21.15.173:0/101180383 shutdown_connections 2024-01-21T19:53:27.580 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.586+0000 7f7d90429700 1 --2- 172.21.15.173:0/101180383 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d880f6510 0x7f7d880f6930 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:27.580 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.586+0000 7f7d90429700 1 -- 172.21.15.173:0/101180383 >> 172.21.15.173:0/101180383 conn(0x7f7d880f1c70 msgr2=0x7f7d880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:27.580 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.586+0000 7f7d90429700 1 -- 172.21.15.173:0/101180383 shutdown_connections 2024-01-21T19:53:27.580 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.586+0000 7f7d90429700 1 -- 172.21.15.173:0/101180383 wait complete. 2024-01-21T19:53:27.580 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.586+0000 7f7d90429700 1 Processor -- start 2024-01-21T19:53:27.580 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.586+0000 7f7d90429700 1 -- start start 2024-01-21T19:53:27.580 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.587+0000 7f7d90429700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d880f6510 0x7f7d88106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:27.580 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.587+0000 7f7d90429700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d88106c20 con 0x7f7d880f6510 2024-01-21T19:53:27.580 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.587+0000 7f7d8e1c5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d880f6510 0x7f7d88106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:27.580 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.587+0000 7f7d8e1c5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d880f6510 0x7f7d88106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:33620/0 (socket says 172.21.15.173:33620) 2024-01-21T19:53:27.580 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.587+0000 7f7d8e1c5700 1 -- 172.21.15.173:0/2926805323 learned_addr learned my addr 172.21.15.173:0/2926805323 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:53:27.580 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.587+0000 7f7d8e1c5700 1 -- 172.21.15.173:0/2926805323 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d78005040 con 0x7f7d880f6510 2024-01-21T19:53:27.581 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.587+0000 7f7d8e1c5700 1 --2- 172.21.15.173:0/2926805323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d880f6510 0x7f7d88106730 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f7d78014000 tx=0x7f7d7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:27.581 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.587+0000 7f7d7effd700 1 -- 172.21.15.173:0/2926805323 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d780095f0 con 0x7f7d880f6510 2024-01-21T19:53:27.581 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.587+0000 7f7d90429700 1 -- 172.21.15.173:0/2926805323 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d88106e20 con 0x7f7d880f6510 2024-01-21T19:53:27.581 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.587+0000 7f7d7effd700 1 -- 172.21.15.173:0/2926805323 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d78007e70 con 0x7f7d880f6510 2024-01-21T19:53:27.581 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.587+0000 7f7d7effd700 1 -- 172.21.15.173:0/2926805323 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d78014ea0 con 0x7f7d880f6510 2024-01-21T19:53:27.581 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.588+0000 7f7d90429700 1 -- 172.21.15.173:0/2926805323 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d881072b0 con 0x7f7d880f6510 2024-01-21T19:53:27.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.588+0000 7f7d7effd700 1 -- 172.21.15.173:0/2926805323 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7d78025070 con 0x7f7d880f6510 2024-01-21T19:53:27.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.588+0000 7f7d90429700 1 -- 172.21.15.173:0/2926805323 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d88045ad0 con 0x7f7d880f6510 2024-01-21T19:53:27.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.588+0000 7f7d7effd700 1 --2- 172.21.15.173:0/2926805323 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f7d7402ff60 0x7f7d74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:27.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.588+0000 7f7d7effd700 1 -- 172.21.15.173:0/2926805323 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7d78040970 con 0x7f7d880f6510 2024-01-21T19:53:27.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.589+0000 7f7d8d9c4700 1 --2- 172.21.15.173:0/2926805323 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f7d7402ff60 0x7f7d74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:27.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.589+0000 7f7d8d9c4700 1 --2- 172.21.15.173:0/2926805323 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f7d7402ff60 0x7f7d74032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f7d840096a0 tx=0x7f7d84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:27.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.593+0000 7f7d7effd700 1 -- 172.21.15.173:0/2926805323 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d780079c0 con 0x7f7d880f6510 2024-01-21T19:53:27.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.801+0000 7f7d90429700 1 -- 172.21.15.173:0/2926805323 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d880fbf60 con 0x7f7d880f6510 2024-01-21T19:53:27.793 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.801+0000 7f7d7effd700 1 -- 172.21.15.173:0/2926805323 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7d7800f300 con 0x7f7d880f6510 2024-01-21T19:53:27.793 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:53:27.793 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:53:27.795 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.803+0000 7f7d90429700 1 -- 172.21.15.173:0/2926805323 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f7d7402ff60 msgr2=0x7f7d74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:27.795 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.803+0000 7f7d90429700 1 --2- 172.21.15.173:0/2926805323 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f7d7402ff60 0x7f7d74032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f7d840096a0 tx=0x7f7d84006b40).stop 2024-01-21T19:53:27.795 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.803+0000 7f7d90429700 1 -- 172.21.15.173:0/2926805323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d880f6510 msgr2=0x7f7d88106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:27.795 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.803+0000 7f7d90429700 1 --2- 172.21.15.173:0/2926805323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d880f6510 0x7f7d88106730 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f7d78014000 tx=0x7f7d7800b040).stop 2024-01-21T19:53:27.795 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.803+0000 7f7d90429700 1 -- 172.21.15.173:0/2926805323 shutdown_connections 2024-01-21T19:53:27.795 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.803+0000 7f7d90429700 1 --2- 172.21.15.173:0/2926805323 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f7d7402ff60 0x7f7d74032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:27.795 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.803+0000 7f7d90429700 1 --2- 172.21.15.173:0/2926805323 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7d880f6510 0x7f7d88106730 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:27.795 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.803+0000 7f7d90429700 1 -- 172.21.15.173:0/2926805323 >> 172.21.15.173:0/2926805323 conn(0x7f7d880f1c70 msgr2=0x7f7d880f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:27.795 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.803+0000 7f7d90429700 1 -- 172.21.15.173:0/2926805323 shutdown_connections 2024-01-21T19:53:27.795 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:27.803+0000 7f7d90429700 1 -- 172.21.15.173:0/2926805323 wait complete. 2024-01-21T19:53:27.796 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:28.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:28 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:26.979104+0000 mgr.smithi169.espgue (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:28.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:28 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:27.802112+0000 mon.smithi169 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.173:0/2926805323' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:53:29.188 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:53:29.188 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:53:30.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.411+0000 7f245d971700 1 -- 172.21.15.173:0/3967224514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24580f6510 msgr2=0x7f24580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:30.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.411+0000 7f245d971700 1 --2- 172.21.15.173:0/3967224514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24580f6510 0x7f24580f6930 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f244c009fb0 tx=0x7f244c009b30).stop 2024-01-21T19:53:30.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.411+0000 7f245d971700 1 -- 172.21.15.173:0/3967224514 shutdown_connections 2024-01-21T19:53:30.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.411+0000 7f245d971700 1 --2- 172.21.15.173:0/3967224514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24580f6510 0x7f24580f6930 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:30.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.411+0000 7f245d971700 1 -- 172.21.15.173:0/3967224514 >> 172.21.15.173:0/3967224514 conn(0x7f24580f1c70 msgr2=0x7f24580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:30.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.411+0000 7f245d971700 1 -- 172.21.15.173:0/3967224514 shutdown_connections 2024-01-21T19:53:30.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.411+0000 7f245d971700 1 -- 172.21.15.173:0/3967224514 wait complete. 2024-01-21T19:53:30.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.412+0000 7f245d971700 1 Processor -- start 2024-01-21T19:53:30.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.412+0000 7f245d971700 1 -- start start 2024-01-21T19:53:30.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.412+0000 7f245d971700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24580f6510 0x7f245806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:30.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.412+0000 7f245d971700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2458069e20 con 0x7f24580f6510 2024-01-21T19:53:30.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.412+0000 7f2456ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24580f6510 0x7f245806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:30.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.412+0000 7f2456ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24580f6510 0x7f245806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:33638/0 (socket says 172.21.15.173:33638) 2024-01-21T19:53:30.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.412+0000 7f2456ffd700 1 -- 172.21.15.173:0/1689907017 learned_addr learned my addr 172.21.15.173:0/1689907017 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:53:30.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.413+0000 7f2456ffd700 1 -- 172.21.15.173:0/1689907017 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f244c005040 con 0x7f24580f6510 2024-01-21T19:53:30.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.413+0000 7f2456ffd700 1 --2- 172.21.15.173:0/1689907017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24580f6510 0x7f245806b9a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f244c013510 tx=0x7f244c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:30.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.413+0000 7f243f7fe700 1 -- 172.21.15.173:0/1689907017 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f244c009320 con 0x7f24580f6510 2024-01-21T19:53:30.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.413+0000 7f245d971700 1 -- 172.21.15.173:0/1689907017 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2458069fc0 con 0x7f24580f6510 2024-01-21T19:53:30.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.413+0000 7f243f7fe700 1 -- 172.21.15.173:0/1689907017 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f244c013f00 con 0x7f24580f6510 2024-01-21T19:53:30.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.413+0000 7f243f7fe700 1 -- 172.21.15.173:0/1689907017 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f244c014d90 con 0x7f24580f6510 2024-01-21T19:53:30.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.413+0000 7f245d971700 1 -- 172.21.15.173:0/1689907017 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f245806a450 con 0x7f24580f6510 2024-01-21T19:53:30.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.414+0000 7f243f7fe700 1 -- 172.21.15.173:0/1689907017 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f244c0263a0 con 0x7f24580f6510 2024-01-21T19:53:30.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.414+0000 7f243f7fe700 1 --2- 172.21.15.173:0/1689907017 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f244002ffb0 0x7f2440032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:30.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.414+0000 7f245d971700 1 -- 172.21.15.173:0/1689907017 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2458059100 con 0x7f24580f6510 2024-01-21T19:53:30.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.414+0000 7f243f7fe700 1 -- 172.21.15.173:0/1689907017 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f244c041320 con 0x7f24580f6510 2024-01-21T19:53:30.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.414+0000 7f24567fc700 1 --2- 172.21.15.173:0/1689907017 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f244002ffb0 0x7f2440032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:30.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.415+0000 7f24567fc700 1 --2- 172.21.15.173:0/1689907017 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f244002ffb0 0x7f2440032470 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f24480096a0 tx=0x7f2448006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:30.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.419+0000 7f243f7fe700 1 -- 172.21.15.173:0/1689907017 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f244c0126f0 con 0x7f24580f6510 2024-01-21T19:53:30.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.624+0000 7f245d971700 1 -- 172.21.15.173:0/1689907017 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24580fbf60 con 0x7f24580f6510 2024-01-21T19:53:30.616 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.624+0000 7f243f7fe700 1 -- 172.21.15.173:0/1689907017 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f244c012330 con 0x7f24580f6510 2024-01-21T19:53:30.616 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:53:30.617 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:53:30.618 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.626+0000 7f245d971700 1 -- 172.21.15.173:0/1689907017 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f244002ffb0 msgr2=0x7f2440032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:30.618 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.626+0000 7f245d971700 1 --2- 172.21.15.173:0/1689907017 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f244002ffb0 0x7f2440032470 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f24480096a0 tx=0x7f2448006b40).stop 2024-01-21T19:53:30.618 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.626+0000 7f245d971700 1 -- 172.21.15.173:0/1689907017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24580f6510 msgr2=0x7f245806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:30.618 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.626+0000 7f245d971700 1 --2- 172.21.15.173:0/1689907017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24580f6510 0x7f245806b9a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f244c013510 tx=0x7f244c00b040).stop 2024-01-21T19:53:30.618 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.626+0000 7f245d971700 1 -- 172.21.15.173:0/1689907017 shutdown_connections 2024-01-21T19:53:30.618 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.626+0000 7f245d971700 1 --2- 172.21.15.173:0/1689907017 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f244002ffb0 0x7f2440032470 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:30.618 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.626+0000 7f245d971700 1 --2- 172.21.15.173:0/1689907017 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f24580f6510 0x7f245806b9a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:30.618 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.626+0000 7f245d971700 1 -- 172.21.15.173:0/1689907017 >> 172.21.15.173:0/1689907017 conn(0x7f24580f1c70 msgr2=0x7f24580f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:30.618 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.626+0000 7f245d971700 1 -- 172.21.15.173:0/1689907017 shutdown_connections 2024-01-21T19:53:30.618 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:30.626+0000 7f245d971700 1 -- 172.21.15.173:0/1689907017 wait complete. 2024-01-21T19:53:30.619 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:31.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:30 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:28.979453+0000 mgr.smithi169.espgue (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:31.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:30 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:30.625207+0000 mon.smithi169 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.173:0/1689907017' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:53:32.034 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:53:32.035 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:53:32.167 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:31.073720+0000 mon.smithi169 (mon.0) 221 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/mirror_snapshot_schedule"}]: dispatch 2024-01-21T19:53:32.167 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:31.077405+0000 mon.smithi169 (mon.0) 222 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/trash_purge_schedule"}]: dispatch 2024-01-21T19:53:33.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:32 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:30.979805+0000 mgr.smithi169.espgue (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:33.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.281+0000 7f69827d7700 1 -- 172.21.15.173:0/3911766277 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f697c0f6510 msgr2=0x7f697c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:33.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.281+0000 7f69827d7700 1 --2- 172.21.15.173:0/3911766277 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f697c0f6510 0x7f697c0f6930 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f696c009fb0 tx=0x7f696c009b30).stop 2024-01-21T19:53:33.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.281+0000 7f69827d7700 1 -- 172.21.15.173:0/3911766277 shutdown_connections 2024-01-21T19:53:33.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.281+0000 7f69827d7700 1 --2- 172.21.15.173:0/3911766277 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f697c0f6510 0x7f697c0f6930 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:33.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.281+0000 7f69827d7700 1 -- 172.21.15.173:0/3911766277 >> 172.21.15.173:0/3911766277 conn(0x7f697c0f1c70 msgr2=0x7f697c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:33.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.281+0000 7f69827d7700 1 -- 172.21.15.173:0/3911766277 shutdown_connections 2024-01-21T19:53:33.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.281+0000 7f69827d7700 1 -- 172.21.15.173:0/3911766277 wait complete. 2024-01-21T19:53:33.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.281+0000 7f69827d7700 1 Processor -- start 2024-01-21T19:53:33.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.282+0000 7f69827d7700 1 -- start start 2024-01-21T19:53:33.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.282+0000 7f69827d7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f697c0f6510 0x7f697c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:33.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.282+0000 7f69827d7700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697c106c20 con 0x7f697c0f6510 2024-01-21T19:53:33.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.282+0000 7f697bfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f697c0f6510 0x7f697c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:33.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.282+0000 7f697bfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f697c0f6510 0x7f697c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:33656/0 (socket says 172.21.15.173:33656) 2024-01-21T19:53:33.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.282+0000 7f697bfff700 1 -- 172.21.15.173:0/1755427507 learned_addr learned my addr 172.21.15.173:0/1755427507 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:53:33.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.282+0000 7f697bfff700 1 -- 172.21.15.173:0/1755427507 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c005040 con 0x7f697c0f6510 2024-01-21T19:53:33.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.282+0000 7f697bfff700 1 --2- 172.21.15.173:0/1755427507 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f697c0f6510 0x7f697c106730 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f696c013510 tx=0x7f696c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:33.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.283+0000 7f6978ff9700 1 -- 172.21.15.173:0/1755427507 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f696c009320 con 0x7f697c0f6510 2024-01-21T19:53:33.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.283+0000 7f6978ff9700 1 -- 172.21.15.173:0/1755427507 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f696c013f00 con 0x7f697c0f6510 2024-01-21T19:53:33.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.283+0000 7f69827d7700 1 -- 172.21.15.173:0/1755427507 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f697c106e20 con 0x7f697c0f6510 2024-01-21T19:53:33.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.283+0000 7f6978ff9700 1 -- 172.21.15.173:0/1755427507 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f696c014d90 con 0x7f697c0f6510 2024-01-21T19:53:33.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.283+0000 7f69827d7700 1 -- 172.21.15.173:0/1755427507 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f697c1072b0 con 0x7f697c0f6510 2024-01-21T19:53:33.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.283+0000 7f6978ff9700 1 -- 172.21.15.173:0/1755427507 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f696c014420 con 0x7f697c0f6510 2024-01-21T19:53:33.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.283+0000 7f6978ff9700 1 --2- 172.21.15.173:0/1755427507 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f696402ff60 0x7f6964032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:33.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.283+0000 7f6978ff9700 1 -- 172.21.15.173:0/1755427507 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f696c041000 con 0x7f697c0f6510 2024-01-21T19:53:33.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.284+0000 7f69827d7700 1 -- 172.21.15.173:0/1755427507 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f697c045ad0 con 0x7f697c0f6510 2024-01-21T19:53:33.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.284+0000 7f697b7fe700 1 --2- 172.21.15.173:0/1755427507 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f696402ff60 0x7f6964032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:33.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.284+0000 7f697b7fe700 1 --2- 172.21.15.173:0/1755427507 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f696402ff60 0x7f6964032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f69700096a0 tx=0x7f6970006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:33.280 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.288+0000 7f6978ff9700 1 -- 172.21.15.173:0/1755427507 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f696c0095f0 con 0x7f697c0f6510 2024-01-21T19:53:33.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.494+0000 7f69827d7700 1 -- 172.21.15.173:0/1755427507 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f697c0fc050 con 0x7f697c0f6510 2024-01-21T19:53:33.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.495+0000 7f6978ff9700 1 -- 172.21.15.173:0/1755427507 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f696c01db70 con 0x7f697c0f6510 2024-01-21T19:53:33.487 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:53:33.487 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:53:33.488 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.497+0000 7f69827d7700 1 -- 172.21.15.173:0/1755427507 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f696402ff60 msgr2=0x7f6964032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:33.488 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.497+0000 7f69827d7700 1 --2- 172.21.15.173:0/1755427507 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f696402ff60 0x7f6964032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f69700096a0 tx=0x7f6970006b40).stop 2024-01-21T19:53:33.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.497+0000 7f69827d7700 1 -- 172.21.15.173:0/1755427507 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f697c0f6510 msgr2=0x7f697c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:33.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.497+0000 7f69827d7700 1 --2- 172.21.15.173:0/1755427507 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f697c0f6510 0x7f697c106730 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f696c013510 tx=0x7f696c00b040).stop 2024-01-21T19:53:33.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.497+0000 7f69827d7700 1 -- 172.21.15.173:0/1755427507 shutdown_connections 2024-01-21T19:53:33.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.497+0000 7f69827d7700 1 --2- 172.21.15.173:0/1755427507 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f696402ff60 0x7f6964032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:33.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.497+0000 7f69827d7700 1 --2- 172.21.15.173:0/1755427507 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f697c0f6510 0x7f697c106730 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:33.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.497+0000 7f69827d7700 1 -- 172.21.15.173:0/1755427507 >> 172.21.15.173:0/1755427507 conn(0x7f697c0f1c70 msgr2=0x7f697c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:33.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.497+0000 7f69827d7700 1 -- 172.21.15.173:0/1755427507 shutdown_connections 2024-01-21T19:53:33.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:33.497+0000 7f69827d7700 1 -- 172.21.15.173:0/1755427507 wait complete. 2024-01-21T19:53:33.489 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:34.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:33 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:33.495848+0000 mon.smithi169 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.173:0/1755427507' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:53:34.900 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:53:34.900 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:53:35.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:34 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:32.980157+0000 mgr.smithi169.espgue (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:36.203 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.209+0000 7f3002c0d700 1 -- 172.21.15.173:0/2145980484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ffc0f6510 msgr2=0x7f2ffc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:36.203 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.209+0000 7f3002c0d700 1 --2- 172.21.15.173:0/2145980484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ffc0f6510 0x7f2ffc0f6930 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f2ff0009fb0 tx=0x7f2ff0009b30).stop 2024-01-21T19:53:36.203 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.209+0000 7f3002c0d700 1 -- 172.21.15.173:0/2145980484 shutdown_connections 2024-01-21T19:53:36.203 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.209+0000 7f3002c0d700 1 --2- 172.21.15.173:0/2145980484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ffc0f6510 0x7f2ffc0f6930 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:36.203 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.209+0000 7f3002c0d700 1 -- 172.21.15.173:0/2145980484 >> 172.21.15.173:0/2145980484 conn(0x7f2ffc0f1c70 msgr2=0x7f2ffc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:36.203 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.209+0000 7f3002c0d700 1 -- 172.21.15.173:0/2145980484 shutdown_connections 2024-01-21T19:53:36.203 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.209+0000 7f3002c0d700 1 -- 172.21.15.173:0/2145980484 wait complete. 2024-01-21T19:53:36.203 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.210+0000 7f3002c0d700 1 Processor -- start 2024-01-21T19:53:36.203 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.210+0000 7f3002c0d700 1 -- start start 2024-01-21T19:53:36.203 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.210+0000 7f3002c0d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ffc0f6510 0x7f2ffc1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:36.203 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.210+0000 7f3002c0d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc106bc0 con 0x7f2ffc0f6510 2024-01-21T19:53:36.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.211+0000 7f30009a9700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ffc0f6510 0x7f2ffc1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:36.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.211+0000 7f30009a9700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ffc0f6510 0x7f2ffc1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:42928/0 (socket says 172.21.15.173:42928) 2024-01-21T19:53:36.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.211+0000 7f30009a9700 1 -- 172.21.15.173:0/1081132214 learned_addr learned my addr 172.21.15.173:0/1081132214 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:53:36.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.211+0000 7f30009a9700 1 -- 172.21.15.173:0/1081132214 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff0005040 con 0x7f2ffc0f6510 2024-01-21T19:53:36.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.211+0000 7f30009a9700 1 --2- 172.21.15.173:0/1081132214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ffc0f6510 0x7f2ffc1066d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f2ff0014000 tx=0x7f2ff000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:36.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.211+0000 7f2ff97fa700 1 -- 172.21.15.173:0/1081132214 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ff00095f0 con 0x7f2ffc0f6510 2024-01-21T19:53:36.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.211+0000 7f3002c0d700 1 -- 172.21.15.173:0/1081132214 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ffc106dc0 con 0x7f2ffc0f6510 2024-01-21T19:53:36.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.211+0000 7f3002c0d700 1 -- 172.21.15.173:0/1081132214 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ffc107250 con 0x7f2ffc0f6510 2024-01-21T19:53:36.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.211+0000 7f2ff97fa700 1 -- 172.21.15.173:0/1081132214 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ff0007e70 con 0x7f2ffc0f6510 2024-01-21T19:53:36.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.211+0000 7f2ff97fa700 1 -- 172.21.15.173:0/1081132214 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ff0014e50 con 0x7f2ffc0f6510 2024-01-21T19:53:36.206 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.212+0000 7f2ff97fa700 1 -- 172.21.15.173:0/1081132214 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2ff0025070 con 0x7f2ffc0f6510 2024-01-21T19:53:36.206 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.212+0000 7f3002c0d700 1 -- 172.21.15.173:0/1081132214 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ffc045ad0 con 0x7f2ffc0f6510 2024-01-21T19:53:36.206 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.212+0000 7f2ff97fa700 1 --2- 172.21.15.173:0/1081132214 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2fec02ff10 0x7f2fec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:36.206 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.212+0000 7f2ff97fa700 1 -- 172.21.15.173:0/1081132214 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2ff0040ad0 con 0x7f2ffc0f6510 2024-01-21T19:53:36.206 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.213+0000 7f2ffbfff700 1 --2- 172.21.15.173:0/1081132214 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2fec02ff10 0x7f2fec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:36.206 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.213+0000 7f2ffbfff700 1 --2- 172.21.15.173:0/1081132214 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2fec02ff10 0x7f2fec0323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f2fe80096a0 tx=0x7f2fe8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:36.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.217+0000 7f2ff97fa700 1 -- 172.21.15.173:0/1081132214 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2ff00079c0 con 0x7f2ffc0f6510 2024-01-21T19:53:36.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.422+0000 7f3002c0d700 1 -- 172.21.15.173:0/1081132214 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ffc108050 con 0x7f2ffc0f6510 2024-01-21T19:53:36.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.423+0000 7f2ff97fa700 1 -- 172.21.15.173:0/1081132214 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2ff000f300 con 0x7f2ffc0f6510 2024-01-21T19:53:36.416 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:53:36.416 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:53:36.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.425+0000 7f3002c0d700 1 -- 172.21.15.173:0/1081132214 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2fec02ff10 msgr2=0x7f2fec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:36.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.425+0000 7f3002c0d700 1 --2- 172.21.15.173:0/1081132214 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2fec02ff10 0x7f2fec0323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f2fe80096a0 tx=0x7f2fe8006b40).stop 2024-01-21T19:53:36.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.425+0000 7f3002c0d700 1 -- 172.21.15.173:0/1081132214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ffc0f6510 msgr2=0x7f2ffc1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:36.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.425+0000 7f3002c0d700 1 --2- 172.21.15.173:0/1081132214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ffc0f6510 0x7f2ffc1066d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f2ff0014000 tx=0x7f2ff000b040).stop 2024-01-21T19:53:36.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.425+0000 7f3002c0d700 1 -- 172.21.15.173:0/1081132214 shutdown_connections 2024-01-21T19:53:36.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.425+0000 7f3002c0d700 1 --2- 172.21.15.173:0/1081132214 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2fec02ff10 0x7f2fec0323d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:36.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.425+0000 7f3002c0d700 1 --2- 172.21.15.173:0/1081132214 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ffc0f6510 0x7f2ffc1066d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:36.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.425+0000 7f3002c0d700 1 -- 172.21.15.173:0/1081132214 >> 172.21.15.173:0/1081132214 conn(0x7f2ffc0f1c70 msgr2=0x7f2ffc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:36.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.425+0000 7f3002c0d700 1 -- 172.21.15.173:0/1081132214 shutdown_connections 2024-01-21T19:53:36.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:36.425+0000 7f3002c0d700 1 -- 172.21.15.173:0/1081132214 wait complete. 2024-01-21T19:53:36.418 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:37.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:36 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:34.980494+0000 mgr.smithi169.espgue (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:37.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:36 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:36.424879+0000 mon.smithi169 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.173:0/1081132214' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:53:37.837 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:53:37.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:53:38.885 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:36.980831+0000 mgr.smithi169.espgue (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:38.886 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cephadm 2024-01-21T19:53:37.899857+0000 mgr.smithi169.espgue (mgr.14162) 155 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi169 ... 2024-01-21T19:53:38.886 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-21T19:53:38.886 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T19:53:38.886 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: stat: stderr time="2024-01-21T19:49:16Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T19:53:38.886 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: stat: stderr time="2024-01-21T19:49:53Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T19:53:38.886 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: stat: stderr time="2024-01-21T19:50:31Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T19:53:38.886 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T19:53:38.886 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-21T19:53:38.886 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T19:53:38.886 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: stat: stderr time="2024-01-21T19:51:45Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T19:53:38.886 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: stat: stderr time="2024-01-21T19:52:23Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T19:53:38.887 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: stat: stderr time="2024-01-21T19:53:00Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T19:53:38.887 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T19:53:38.887 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Traceback (most recent call last): 2024-01-21T19:53:38.887 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T19:53:38.887 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: main() 2024-01-21T19:53:38.887 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T19:53:38.887 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: r = ctx.func(ctx) 2024-01-21T19:53:38.887 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T19:53:38.887 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: return func(ctx) 2024-01-21T19:53:38.887 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T19:53:38.887 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T19:53:38.887 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-01-21T19:53:38.887 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-21T19:53:38.887 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T19:53:38.888 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: raise RuntimeError('uid/gid not found') 2024-01-21T19:53:38.888 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: RuntimeError: uid/gid not found 2024-01-21T19:53:38.888 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Traceback (most recent call last): 2024-01-21T19:53:38.888 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T19:53:38.888 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: yield (conn, connr) 2024-01-21T19:53:38.888 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T19:53:38.888 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: code, '\n'.join(err))) 2024-01-21T19:53:38.888 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi169 ... 2024-01-21T19:53:38.888 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-21T19:53:38.888 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T19:53:38.888 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: stat: stderr time="2024-01-21T19:49:16Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T19:53:38.888 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: stat: stderr time="2024-01-21T19:49:53Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T19:53:38.888 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: stat: stderr time="2024-01-21T19:50:31Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T19:53:38.889 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T19:53:38.889 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi169 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-21T19:53:38.889 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T19:53:38.889 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: stat: stderr time="2024-01-21T19:51:45Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T19:53:38.889 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: stat: stderr time="2024-01-21T19:52:23Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T19:53:38.889 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: stat: stderr time="2024-01-21T19:53:00Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T19:53:38.889 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T19:53:38.889 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Traceback (most recent call last): 2024-01-21T19:53:38.889 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T19:53:38.889 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: main() 2024-01-21T19:53:38.889 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T19:53:38.889 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: r = ctx.func(ctx) 2024-01-21T19:53:38.889 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T19:53:38.889 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: return func(ctx) 2024-01-21T19:53:38.890 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T19:53:38.890 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T19:53:38.890 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-01-21T19:53:38.890 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-21T19:53:38.890 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T19:53:38.890 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: raise RuntimeError('uid/gid not found') 2024-01-21T19:53:38.890 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: RuntimeError: uid/gid not found 2024-01-21T19:53:38.890 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:37.901038+0000 mon.smithi169 (mon.0) 225 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi169", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-21T19:53:38.890 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:37.904198+0000 mon.smithi169 (mon.0) 226 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi169", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-21T19:53:38.890 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:37.904953+0000 mon.smithi169 (mon.0) 227 : audit [DBG] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T19:53:38.890 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cephadm 2024-01-21T19:53:37.905716+0000 mgr.smithi169.espgue (mgr.14162) 156 : cephadm [INF] Deploying daemon crash.smithi169 on smithi169 2024-01-21T19:53:39.081 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.088+0000 7f8cc14d6700 1 -- 172.21.15.173:0/2135868474 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cbc060280 msgr2=0x7f8cbc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:39.081 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.088+0000 7f8cc14d6700 1 --2- 172.21.15.173:0/2135868474 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cbc060280 0x7f8cbc0f90f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f8cac009fb0 tx=0x7f8cac009b30).stop 2024-01-21T19:53:39.081 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.088+0000 7f8cc14d6700 1 -- 172.21.15.173:0/2135868474 shutdown_connections 2024-01-21T19:53:39.081 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.088+0000 7f8cc14d6700 1 --2- 172.21.15.173:0/2135868474 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cbc060280 0x7f8cbc0f90f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:39.081 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.088+0000 7f8cc14d6700 1 -- 172.21.15.173:0/2135868474 >> 172.21.15.173:0/2135868474 conn(0x7f8cbc0f0b70 msgr2=0x7f8cbc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:39.081 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.088+0000 7f8cc14d6700 1 -- 172.21.15.173:0/2135868474 shutdown_connections 2024-01-21T19:53:39.081 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.088+0000 7f8cc14d6700 1 -- 172.21.15.173:0/2135868474 wait complete. 2024-01-21T19:53:39.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.089+0000 7f8cc14d6700 1 Processor -- start 2024-01-21T19:53:39.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.089+0000 7f8cc14d6700 1 -- start start 2024-01-21T19:53:39.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.089+0000 7f8cc14d6700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cbc060280 0x7f8cbc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:39.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.089+0000 7f8cc14d6700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc106bd0 con 0x7f8cbc060280 2024-01-21T19:53:39.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.089+0000 7f8cbaffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cbc060280 0x7f8cbc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:39.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.089+0000 7f8cbaffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cbc060280 0x7f8cbc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:42938/0 (socket says 172.21.15.173:42938) 2024-01-21T19:53:39.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.089+0000 7f8cbaffd700 1 -- 172.21.15.173:0/4046953205 learned_addr learned my addr 172.21.15.173:0/4046953205 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:53:39.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.089+0000 7f8cbaffd700 1 -- 172.21.15.173:0/4046953205 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac005040 con 0x7f8cbc060280 2024-01-21T19:53:39.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.090+0000 7f8cbaffd700 1 --2- 172.21.15.173:0/4046953205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cbc060280 0x7f8cbc1066e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f8cac013510 tx=0x7f8cac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:39.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.090+0000 7f8cb37fe700 1 -- 172.21.15.173:0/4046953205 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cac009320 con 0x7f8cbc060280 2024-01-21T19:53:39.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.090+0000 7f8cb37fe700 1 -- 172.21.15.173:0/4046953205 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cac013f00 con 0x7f8cbc060280 2024-01-21T19:53:39.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.090+0000 7f8cc14d6700 1 -- 172.21.15.173:0/4046953205 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cbc106dd0 con 0x7f8cbc060280 2024-01-21T19:53:39.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.090+0000 7f8cb37fe700 1 -- 172.21.15.173:0/4046953205 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cac014dc0 con 0x7f8cbc060280 2024-01-21T19:53:39.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.090+0000 7f8cc14d6700 1 -- 172.21.15.173:0/4046953205 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cbc107260 con 0x7f8cbc060280 2024-01-21T19:53:39.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.090+0000 7f8cb37fe700 1 -- 172.21.15.173:0/4046953205 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8cac01d3d0 con 0x7f8cbc060280 2024-01-21T19:53:39.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.091+0000 7f8cb37fe700 1 --2- 172.21.15.173:0/4046953205 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8ca802ff60 0x7f8ca8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:39.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.091+0000 7f8cc14d6700 1 -- 172.21.15.173:0/4046953205 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cbc045ad0 con 0x7f8cbc060280 2024-01-21T19:53:39.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.091+0000 7f8cb37fe700 1 -- 172.21.15.173:0/4046953205 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8cac041120 con 0x7f8cbc060280 2024-01-21T19:53:39.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.091+0000 7f8cba7fc700 1 --2- 172.21.15.173:0/4046953205 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8ca802ff60 0x7f8ca8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:39.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.091+0000 7f8cba7fc700 1 --2- 172.21.15.173:0/4046953205 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8ca802ff60 0x7f8ca8032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f8ca40096a0 tx=0x7f8ca4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:39.087 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.095+0000 7f8cb37fe700 1 -- 172.21.15.173:0/4046953205 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8cac01db80 con 0x7f8cbc060280 2024-01-21T19:53:39.292 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.299+0000 7f8cc14d6700 1 -- 172.21.15.173:0/4046953205 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8cbc108110 con 0x7f8cbc060280 2024-01-21T19:53:39.292 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.300+0000 7f8cb37fe700 1 -- 172.21.15.173:0/4046953205 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8cac02c090 con 0x7f8cbc060280 2024-01-21T19:53:39.293 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:53:39.293 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:53:39.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.301+0000 7f8cc14d6700 1 -- 172.21.15.173:0/4046953205 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8ca802ff60 msgr2=0x7f8ca8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:39.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.301+0000 7f8cc14d6700 1 --2- 172.21.15.173:0/4046953205 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8ca802ff60 0x7f8ca8032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f8ca40096a0 tx=0x7f8ca4006b40).stop 2024-01-21T19:53:39.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.301+0000 7f8cc14d6700 1 -- 172.21.15.173:0/4046953205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cbc060280 msgr2=0x7f8cbc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:39.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.301+0000 7f8cc14d6700 1 --2- 172.21.15.173:0/4046953205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cbc060280 0x7f8cbc1066e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f8cac013510 tx=0x7f8cac00b040).stop 2024-01-21T19:53:39.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.301+0000 7f8cc14d6700 1 -- 172.21.15.173:0/4046953205 shutdown_connections 2024-01-21T19:53:39.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.301+0000 7f8cc14d6700 1 --2- 172.21.15.173:0/4046953205 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8ca802ff60 0x7f8ca8032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:39.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.301+0000 7f8cc14d6700 1 --2- 172.21.15.173:0/4046953205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8cbc060280 0x7f8cbc1066e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:39.295 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.301+0000 7f8cc14d6700 1 -- 172.21.15.173:0/4046953205 >> 172.21.15.173:0/4046953205 conn(0x7f8cbc0f0b70 msgr2=0x7f8cbc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:39.295 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.301+0000 7f8cc14d6700 1 -- 172.21.15.173:0/4046953205 shutdown_connections 2024-01-21T19:53:39.295 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:39.301+0000 7f8cc14d6700 1 -- 172.21.15.173:0/4046953205 wait complete. 2024-01-21T19:53:39.295 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:40.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:39 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:39.301571+0000 mon.smithi169 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.173:0/4046953205' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:53:40.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:39 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:39.388815+0000 mon.smithi169 (mon.0) 229 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' 2024-01-21T19:53:40.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:39 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:39.557060+0000 mon.smithi169 (mon.0) 230 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' 2024-01-21T19:53:40.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:39 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:39.560112+0000 mon.smithi169 (mon.0) 231 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' 2024-01-21T19:53:40.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:39 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:39.560544+0000 mon.smithi169 (mon.0) 232 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-21T19:53:40.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:39 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:39.564062+0000 mon.smithi169 (mon.0) 233 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' 2024-01-21T19:53:40.723 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:53:40.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:53:41.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:40 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:38.981077+0000 mgr.smithi169.espgue (mgr.14162) 157 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:41.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:40 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:39.560977+0000 mgr.smithi169.espgue (mgr.14162) 158 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-21T19:53:41.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:40 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cephadm 2024-01-21T19:53:39.567396+0000 mgr.smithi169.espgue (mgr.14162) 159 : cephadm [INF] Deploying daemon grafana.smithi169 on smithi169 2024-01-21T19:53:41.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.988+0000 7f08297a1700 1 -- 172.21.15.173:0/2107707450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f08240f62a0 msgr2=0x7f08240f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:41.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.988+0000 7f08297a1700 1 --2- 172.21.15.173:0/2107707450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f08240f62a0 0x7f08240f66c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f080c009fb0 tx=0x7f080c009b30).stop 2024-01-21T19:53:41.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.988+0000 7f08297a1700 1 -- 172.21.15.173:0/2107707450 shutdown_connections 2024-01-21T19:53:41.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.988+0000 7f08297a1700 1 --2- 172.21.15.173:0/2107707450 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f08240f62a0 0x7f08240f66c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:41.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.988+0000 7f08297a1700 1 -- 172.21.15.173:0/2107707450 >> 172.21.15.173:0/2107707450 conn(0x7f08240f1aa0 msgr2=0x7f08240f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:41.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.988+0000 7f08297a1700 1 -- 172.21.15.173:0/2107707450 shutdown_connections 2024-01-21T19:53:41.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.988+0000 7f08297a1700 1 -- 172.21.15.173:0/2107707450 wait complete. 2024-01-21T19:53:41.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.989+0000 7f08297a1700 1 Processor -- start 2024-01-21T19:53:41.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.989+0000 7f08297a1700 1 -- start start 2024-01-21T19:53:41.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.989+0000 7f08297a1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f08240f62a0 0x7f08241064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.989+0000 7f08297a1700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08241069a0 con 0x7f08240f62a0 2024-01-21T19:53:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.989+0000 7f0823fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f08240f62a0 0x7f08241064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.989+0000 7f0823fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f08240f62a0 0x7f08241064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:42958/0 (socket says 172.21.15.173:42958) 2024-01-21T19:53:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.989+0000 7f0823fff700 1 -- 172.21.15.173:0/727806352 learned_addr learned my addr 172.21.15.173:0/727806352 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:53:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.989+0000 7f0823fff700 1 -- 172.21.15.173:0/727806352 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080c005040 con 0x7f08240f62a0 2024-01-21T19:53:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.990+0000 7f0823fff700 1 --2- 172.21.15.173:0/727806352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f08240f62a0 0x7f08241064b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f080c013510 tx=0x7f080c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.990+0000 7f0820ff9700 1 -- 172.21.15.173:0/727806352 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f080c009320 con 0x7f08240f62a0 2024-01-21T19:53:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.990+0000 7f08297a1700 1 -- 172.21.15.173:0/727806352 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0824106ba0 con 0x7f08240f62a0 2024-01-21T19:53:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.990+0000 7f0820ff9700 1 -- 172.21.15.173:0/727806352 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f080c013f00 con 0x7f08240f62a0 2024-01-21T19:53:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.990+0000 7f0820ff9700 1 -- 172.21.15.173:0/727806352 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f080c014dc0 con 0x7f08240f62a0 2024-01-21T19:53:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.990+0000 7f08297a1700 1 -- 172.21.15.173:0/727806352 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0824107030 con 0x7f08240f62a0 2024-01-21T19:53:41.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.991+0000 7f0820ff9700 1 -- 172.21.15.173:0/727806352 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f080c014420 con 0x7f08240f62a0 2024-01-21T19:53:41.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.991+0000 7f08297a1700 1 -- 172.21.15.173:0/727806352 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0824045ad0 con 0x7f08240f62a0 2024-01-21T19:53:41.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.991+0000 7f0820ff9700 1 --2- 172.21.15.173:0/727806352 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f081002ff60 0x7f0810032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:41.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.991+0000 7f0820ff9700 1 -- 172.21.15.173:0/727806352 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f080c0410b0 con 0x7f08240f62a0 2024-01-21T19:53:41.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.991+0000 7f08237fe700 1 --2- 172.21.15.173:0/727806352 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f081002ff60 0x7f0810032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:41.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.991+0000 7f08237fe700 1 --2- 172.21.15.173:0/727806352 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f081002ff60 0x7f0810032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f08140096a0 tx=0x7f0814006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:41.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:41.996+0000 7f0820ff9700 1 -- 172.21.15.173:0/727806352 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f080c0095f0 con 0x7f08240f62a0 2024-01-21T19:53:42.193 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:42.200+0000 7f08297a1700 1 -- 172.21.15.173:0/727806352 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0824107ee0 con 0x7f08240f62a0 2024-01-21T19:53:42.194 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:42.201+0000 7f0820ff9700 1 -- 172.21.15.173:0/727806352 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f080c01cb10 con 0x7f08240f62a0 2024-01-21T19:53:42.194 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:53:42.194 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:53:42.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:42.202+0000 7f08297a1700 1 -- 172.21.15.173:0/727806352 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f081002ff60 msgr2=0x7f0810032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:42.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:42.202+0000 7f08297a1700 1 --2- 172.21.15.173:0/727806352 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f081002ff60 0x7f0810032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f08140096a0 tx=0x7f0814006b40).stop 2024-01-21T19:53:42.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:42.202+0000 7f08297a1700 1 -- 172.21.15.173:0/727806352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f08240f62a0 msgr2=0x7f08241064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:42.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:42.202+0000 7f08297a1700 1 --2- 172.21.15.173:0/727806352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f08240f62a0 0x7f08241064b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f080c013510 tx=0x7f080c00b040).stop 2024-01-21T19:53:42.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:42.202+0000 7f08297a1700 1 -- 172.21.15.173:0/727806352 shutdown_connections 2024-01-21T19:53:42.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:42.202+0000 7f08297a1700 1 --2- 172.21.15.173:0/727806352 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f081002ff60 0x7f0810032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:42.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:42.202+0000 7f08297a1700 1 --2- 172.21.15.173:0/727806352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f08240f62a0 0x7f08241064b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:42.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:42.202+0000 7f08297a1700 1 -- 172.21.15.173:0/727806352 >> 172.21.15.173:0/727806352 conn(0x7f08240f1aa0 msgr2=0x7f08240f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:42.202+0000 7f08297a1700 1 -- 172.21.15.173:0/727806352 shutdown_connections 2024-01-21T19:53:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:42.202+0000 7f08297a1700 1 -- 172.21.15.173:0/727806352 wait complete. 2024-01-21T19:53:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:43.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:40.981388+0000 mgr.smithi169.espgue (mgr.14162) 160 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:43.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:42.202611+0000 mon.smithi169 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.173:0/727806352' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:53:43.591 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:53:43.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:53:44.754 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.761+0000 7f2910c42700 1 -- 172.21.15.173:0/4064381844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f290c0f5410 msgr2=0x7f290c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:44.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.761+0000 7f2910c42700 1 --2- 172.21.15.173:0/4064381844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f290c0f5410 0x7f290c0f5830 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f28f4009fb0 tx=0x7f28f4009b30).stop 2024-01-21T19:53:44.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.762+0000 7f2910c42700 1 -- 172.21.15.173:0/4064381844 shutdown_connections 2024-01-21T19:53:44.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.762+0000 7f2910c42700 1 --2- 172.21.15.173:0/4064381844 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f290c0f5410 0x7f290c0f5830 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:44.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.762+0000 7f2910c42700 1 -- 172.21.15.173:0/4064381844 >> 172.21.15.173:0/4064381844 conn(0x7f290c0f0b70 msgr2=0x7f290c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:44.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.762+0000 7f2910c42700 1 -- 172.21.15.173:0/4064381844 shutdown_connections 2024-01-21T19:53:44.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.762+0000 7f2910c42700 1 -- 172.21.15.173:0/4064381844 wait complete. 2024-01-21T19:53:44.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.762+0000 7f2910c42700 1 Processor -- start 2024-01-21T19:53:44.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.762+0000 7f2910c42700 1 -- start start 2024-01-21T19:53:44.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.763+0000 7f2910c42700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f290c0f5410 0x7f290c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:44.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.763+0000 7f2910c42700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f290c105b20 con 0x7f290c0f5410 2024-01-21T19:53:44.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.763+0000 7f290a59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f290c0f5410 0x7f290c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:44.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.763+0000 7f290a59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f290c0f5410 0x7f290c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:40320/0 (socket says 172.21.15.173:40320) 2024-01-21T19:53:44.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.763+0000 7f290a59c700 1 -- 172.21.15.173:0/913074491 learned_addr learned my addr 172.21.15.173:0/913074491 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:53:44.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.763+0000 7f290a59c700 1 -- 172.21.15.173:0/913074491 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f4005040 con 0x7f290c0f5410 2024-01-21T19:53:44.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.763+0000 7f290a59c700 1 --2- 172.21.15.173:0/913074491 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f290c0f5410 0x7f290c105630 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f28f4013510 tx=0x7f28f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:44.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.763+0000 7f2902ffd700 1 -- 172.21.15.173:0/913074491 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28f4009320 con 0x7f290c0f5410 2024-01-21T19:53:44.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.763+0000 7f2902ffd700 1 -- 172.21.15.173:0/913074491 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f28f4013f00 con 0x7f290c0f5410 2024-01-21T19:53:44.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.763+0000 7f2910c42700 1 -- 172.21.15.173:0/913074491 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f290c105d20 con 0x7f290c0f5410 2024-01-21T19:53:44.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.763+0000 7f2902ffd700 1 -- 172.21.15.173:0/913074491 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28f4014d90 con 0x7f290c0f5410 2024-01-21T19:53:44.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.763+0000 7f2910c42700 1 -- 172.21.15.173:0/913074491 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f290c1061b0 con 0x7f290c0f5410 2024-01-21T19:53:44.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.764+0000 7f2902ffd700 1 -- 172.21.15.173:0/913074491 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f28f40095f0 con 0x7f290c0f5410 2024-01-21T19:53:44.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.764+0000 7f2910c42700 1 -- 172.21.15.173:0/913074491 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f290c045ad0 con 0x7f290c0f5410 2024-01-21T19:53:44.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.764+0000 7f2902ffd700 1 --2- 172.21.15.173:0/913074491 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f28f802ffb0 0x7f28f8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:44.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.764+0000 7f2902ffd700 1 -- 172.21.15.173:0/913074491 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f28f40410a0 con 0x7f290c0f5410 2024-01-21T19:53:44.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.765+0000 7f2909d9b700 1 --2- 172.21.15.173:0/913074491 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f28f802ffb0 0x7f28f8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:44.759 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.765+0000 7f2909d9b700 1 --2- 172.21.15.173:0/913074491 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f28f802ffb0 0x7f28f8032470 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f28fc0096a0 tx=0x7f28fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:44.761 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.769+0000 7f2902ffd700 1 -- 172.21.15.173:0/913074491 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f28f406d2b0 con 0x7f290c0f5410 2024-01-21T19:53:44.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.972+0000 7f2910c42700 1 -- 172.21.15.173:0/913074491 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f290c0fae60 con 0x7f290c0f5410 2024-01-21T19:53:44.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.973+0000 7f2902ffd700 1 -- 172.21.15.173:0/913074491 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f28f4014420 con 0x7f290c0f5410 2024-01-21T19:53:44.966 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:53:44.966 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:53:44.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.975+0000 7f2910c42700 1 -- 172.21.15.173:0/913074491 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f28f802ffb0 msgr2=0x7f28f8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:44.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.975+0000 7f2910c42700 1 --2- 172.21.15.173:0/913074491 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f28f802ffb0 0x7f28f8032470 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f28fc0096a0 tx=0x7f28fc006b40).stop 2024-01-21T19:53:44.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.975+0000 7f2910c42700 1 -- 172.21.15.173:0/913074491 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f290c0f5410 msgr2=0x7f290c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:44.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.975+0000 7f2910c42700 1 --2- 172.21.15.173:0/913074491 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f290c0f5410 0x7f290c105630 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f28f4013510 tx=0x7f28f400b040).stop 2024-01-21T19:53:44.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.975+0000 7f2910c42700 1 -- 172.21.15.173:0/913074491 shutdown_connections 2024-01-21T19:53:44.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.975+0000 7f2910c42700 1 --2- 172.21.15.173:0/913074491 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f28f802ffb0 0x7f28f8032470 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:44.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.975+0000 7f2910c42700 1 --2- 172.21.15.173:0/913074491 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f290c0f5410 0x7f290c105630 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:44.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.975+0000 7f2910c42700 1 -- 172.21.15.173:0/913074491 >> 172.21.15.173:0/913074491 conn(0x7f290c0f0b70 msgr2=0x7f290c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:44.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.975+0000 7f2910c42700 1 -- 172.21.15.173:0/913074491 shutdown_connections 2024-01-21T19:53:44.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:44.975+0000 7f2910c42700 1 -- 172.21.15.173:0/913074491 wait complete. 2024-01-21T19:53:44.968 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:45.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:44 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:42.981659+0000 mgr.smithi169.espgue (mgr.14162) 161 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:46.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:45 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:44.974903+0000 mon.smithi169 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.173:0/913074491' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:53:46.361 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:53:46.362 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:53:47.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:46 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:44.982037+0000 mgr.smithi169.espgue (mgr.14162) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:47.531 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.536+0000 7fafce42c700 1 -- 172.21.15.173:0/3203888231 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafc80f62a0 msgr2=0x7fafc80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:47.531 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.536+0000 7fafce42c700 1 --2- 172.21.15.173:0/3203888231 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafc80f62a0 0x7fafc80f66c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fafc4009fb0 tx=0x7fafc4009b30).stop 2024-01-21T19:53:47.531 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.536+0000 7fafce42c700 1 -- 172.21.15.173:0/3203888231 shutdown_connections 2024-01-21T19:53:47.531 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.536+0000 7fafce42c700 1 --2- 172.21.15.173:0/3203888231 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafc80f62a0 0x7fafc80f66c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:47.531 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.536+0000 7fafce42c700 1 -- 172.21.15.173:0/3203888231 >> 172.21.15.173:0/3203888231 conn(0x7fafc80f1aa0 msgr2=0x7fafc80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:47.531 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.536+0000 7fafce42c700 1 -- 172.21.15.173:0/3203888231 shutdown_connections 2024-01-21T19:53:47.531 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.536+0000 7fafce42c700 1 -- 172.21.15.173:0/3203888231 wait complete. 2024-01-21T19:53:47.531 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.537+0000 7fafce42c700 1 Processor -- start 2024-01-21T19:53:47.531 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.537+0000 7fafce42c700 1 -- start start 2024-01-21T19:53:47.531 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.537+0000 7fafce42c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafc80f62a0 0x7fafc81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:47.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.537+0000 7fafce42c700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc81069a0 con 0x7fafc80f62a0 2024-01-21T19:53:47.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.537+0000 7fafcd42a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafc80f62a0 0x7fafc81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:47.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.537+0000 7fafcd42a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafc80f62a0 0x7fafc81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:40344/0 (socket says 172.21.15.173:40344) 2024-01-21T19:53:47.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.537+0000 7fafcd42a700 1 -- 172.21.15.173:0/3233201613 learned_addr learned my addr 172.21.15.173:0/3233201613 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:53:47.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.538+0000 7fafcd42a700 1 -- 172.21.15.173:0/3233201613 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc4005040 con 0x7fafc80f62a0 2024-01-21T19:53:47.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.538+0000 7fafcd42a700 1 --2- 172.21.15.173:0/3233201613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafc80f62a0 0x7fafc81064b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fafc4013510 tx=0x7fafc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:47.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.538+0000 7fafbdffb700 1 -- 172.21.15.173:0/3233201613 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafc4009320 con 0x7fafc80f62a0 2024-01-21T19:53:47.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.538+0000 7fafbdffb700 1 -- 172.21.15.173:0/3233201613 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fafc4013f00 con 0x7fafc80f62a0 2024-01-21T19:53:47.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.538+0000 7fafce42c700 1 -- 172.21.15.173:0/3233201613 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc8106ba0 con 0x7fafc80f62a0 2024-01-21T19:53:47.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.538+0000 7fafbdffb700 1 -- 172.21.15.173:0/3233201613 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafc4014dc0 con 0x7fafc80f62a0 2024-01-21T19:53:47.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.538+0000 7fafce42c700 1 -- 172.21.15.173:0/3233201613 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc8107030 con 0x7fafc80f62a0 2024-01-21T19:53:47.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.539+0000 7fafbdffb700 1 -- 172.21.15.173:0/3233201613 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fafc4014420 con 0x7fafc80f62a0 2024-01-21T19:53:47.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.539+0000 7fafce42c700 1 -- 172.21.15.173:0/3233201613 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc8045ad0 con 0x7fafc80f62a0 2024-01-21T19:53:47.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.539+0000 7fafbdffb700 1 --2- 172.21.15.173:0/3233201613 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fafb402ff60 0x7fafb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:47.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.539+0000 7fafbdffb700 1 -- 172.21.15.173:0/3233201613 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fafc4041020 con 0x7fafc80f62a0 2024-01-21T19:53:47.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.539+0000 7fafccc29700 1 --2- 172.21.15.173:0/3233201613 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fafb402ff60 0x7fafb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:47.534 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.539+0000 7fafccc29700 1 --2- 172.21.15.173:0/3233201613 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fafb402ff60 0x7fafb4032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fafb80096a0 tx=0x7fafb8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:47.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.544+0000 7fafbdffb700 1 -- 172.21.15.173:0/3233201613 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fafc40095f0 con 0x7fafc80f62a0 2024-01-21T19:53:47.742 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.750+0000 7fafce42c700 1 -- 172.21.15.173:0/3233201613 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fafc8107e80 con 0x7fafc80f62a0 2024-01-21T19:53:47.743 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.750+0000 7fafbdffb700 1 -- 172.21.15.173:0/3233201613 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fafc401cb10 con 0x7fafc80f62a0 2024-01-21T19:53:47.743 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:53:47.743 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:53:47.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.752+0000 7fafce42c700 1 -- 172.21.15.173:0/3233201613 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fafb402ff60 msgr2=0x7fafb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:47.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.752+0000 7fafce42c700 1 --2- 172.21.15.173:0/3233201613 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fafb402ff60 0x7fafb4032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fafb80096a0 tx=0x7fafb8006b40).stop 2024-01-21T19:53:47.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.752+0000 7fafce42c700 1 -- 172.21.15.173:0/3233201613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafc80f62a0 msgr2=0x7fafc81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:47.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.752+0000 7fafce42c700 1 --2- 172.21.15.173:0/3233201613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafc80f62a0 0x7fafc81064b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fafc4013510 tx=0x7fafc400b040).stop 2024-01-21T19:53:47.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.752+0000 7fafce42c700 1 -- 172.21.15.173:0/3233201613 shutdown_connections 2024-01-21T19:53:47.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.752+0000 7fafce42c700 1 --2- 172.21.15.173:0/3233201613 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fafb402ff60 0x7fafb4032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:47.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.752+0000 7fafce42c700 1 --2- 172.21.15.173:0/3233201613 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fafc80f62a0 0x7fafc81064b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:47.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.752+0000 7fafce42c700 1 -- 172.21.15.173:0/3233201613 >> 172.21.15.173:0/3233201613 conn(0x7fafc80f1aa0 msgr2=0x7fafc80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:47.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.752+0000 7fafce42c700 1 -- 172.21.15.173:0/3233201613 shutdown_connections 2024-01-21T19:53:47.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:47.752+0000 7fafce42c700 1 -- 172.21.15.173:0/3233201613 wait complete. 2024-01-21T19:53:47.745 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:49.131 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:53:49.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:53:49.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:46.982243+0000 mgr.smithi169.espgue (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:49.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:47.751986+0000 mon.smithi169 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.173:0/3233201613' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:53:50.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.366+0000 7f5753fff700 1 -- 172.21.15.173:0/1756217547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57540f62a0 msgr2=0x7f57540f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:50.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.366+0000 7f5753fff700 1 --2- 172.21.15.173:0/1756217547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57540f62a0 0x7f57540f66c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f5744009fb0 tx=0x7f5744009b30).stop 2024-01-21T19:53:50.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.366+0000 7f5753fff700 1 -- 172.21.15.173:0/1756217547 shutdown_connections 2024-01-21T19:53:50.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.366+0000 7f5753fff700 1 --2- 172.21.15.173:0/1756217547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57540f62a0 0x7f57540f66c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:50.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.366+0000 7f5753fff700 1 -- 172.21.15.173:0/1756217547 >> 172.21.15.173:0/1756217547 conn(0x7f57540f1aa0 msgr2=0x7f57540f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:50.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.366+0000 7f5753fff700 1 -- 172.21.15.173:0/1756217547 shutdown_connections 2024-01-21T19:53:50.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.366+0000 7f5753fff700 1 -- 172.21.15.173:0/1756217547 wait complete. 2024-01-21T19:53:50.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.366+0000 7f5753fff700 1 Processor -- start 2024-01-21T19:53:50.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.366+0000 7f5753fff700 1 -- start start 2024-01-21T19:53:50.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.366+0000 7f5753fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57540f62a0 0x7f57541064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:50.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.366+0000 7f5753fff700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57541069a0 con 0x7f57540f62a0 2024-01-21T19:53:50.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.367+0000 7f5752ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57540f62a0 0x7f57541064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:50.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.367+0000 7f5752ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57540f62a0 0x7f57541064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:40350/0 (socket says 172.21.15.173:40350) 2024-01-21T19:53:50.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.367+0000 7f5752ffd700 1 -- 172.21.15.173:0/4057568765 learned_addr learned my addr 172.21.15.173:0/4057568765 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:53:50.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.367+0000 7f5752ffd700 1 -- 172.21.15.173:0/4057568765 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5744005040 con 0x7f57540f62a0 2024-01-21T19:53:50.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.367+0000 7f5752ffd700 1 --2- 172.21.15.173:0/4057568765 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57540f62a0 0x7f57541064b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f5744013510 tx=0x7f574400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:50.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.367+0000 7f573b7fe700 1 -- 172.21.15.173:0/4057568765 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5744009320 con 0x7f57540f62a0 2024-01-21T19:53:50.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.367+0000 7f573b7fe700 1 -- 172.21.15.173:0/4057568765 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5744013f00 con 0x7f57540f62a0 2024-01-21T19:53:50.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.367+0000 7f5753fff700 1 -- 172.21.15.173:0/4057568765 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5754106ba0 con 0x7f57540f62a0 2024-01-21T19:53:50.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.367+0000 7f573b7fe700 1 -- 172.21.15.173:0/4057568765 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5744014dc0 con 0x7f57540f62a0 2024-01-21T19:53:50.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.367+0000 7f5753fff700 1 -- 172.21.15.173:0/4057568765 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5754107030 con 0x7f57540f62a0 2024-01-21T19:53:50.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.368+0000 7f573b7fe700 1 -- 172.21.15.173:0/4057568765 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f574401d3d0 con 0x7f57540f62a0 2024-01-21T19:53:50.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.368+0000 7f573b7fe700 1 --2- 172.21.15.173:0/4057568765 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f573c02ff60 0x7f573c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:50.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.368+0000 7f573b7fe700 1 -- 172.21.15.173:0/4057568765 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f57440410f0 con 0x7f57540f62a0 2024-01-21T19:53:50.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.368+0000 7f5753fff700 1 -- 172.21.15.173:0/4057568765 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5754045ad0 con 0x7f57540f62a0 2024-01-21T19:53:50.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.368+0000 7f57527fc700 1 --2- 172.21.15.173:0/4057568765 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f573c02ff60 0x7f573c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:50.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.369+0000 7f57527fc700 1 --2- 172.21.15.173:0/4057568765 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f573c02ff60 0x7f573c032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f57480096a0 tx=0x7f5748006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:50.367 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.375+0000 7f573b7fe700 1 -- 172.21.15.173:0/4057568765 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5744012920 con 0x7f57540f62a0 2024-01-21T19:53:50.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.580+0000 7f5753fff700 1 -- 172.21.15.173:0/4057568765 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5754107d50 con 0x7f57540f62a0 2024-01-21T19:53:50.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.580+0000 7f573b7fe700 1 -- 172.21.15.173:0/4057568765 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f574402b020 con 0x7f57540f62a0 2024-01-21T19:53:50.573 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:53:50.574 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:53:50.575 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.582+0000 7f5753fff700 1 -- 172.21.15.173:0/4057568765 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f573c02ff60 msgr2=0x7f573c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:50.575 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.582+0000 7f5753fff700 1 --2- 172.21.15.173:0/4057568765 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f573c02ff60 0x7f573c032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f57480096a0 tx=0x7f5748006b40).stop 2024-01-21T19:53:50.575 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.582+0000 7f5753fff700 1 -- 172.21.15.173:0/4057568765 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57540f62a0 msgr2=0x7f57541064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:50.575 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.582+0000 7f5753fff700 1 --2- 172.21.15.173:0/4057568765 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57540f62a0 0x7f57541064b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f5744013510 tx=0x7f574400b040).stop 2024-01-21T19:53:50.575 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.582+0000 7f5753fff700 1 -- 172.21.15.173:0/4057568765 shutdown_connections 2024-01-21T19:53:50.575 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.582+0000 7f5753fff700 1 --2- 172.21.15.173:0/4057568765 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f573c02ff60 0x7f573c032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:50.576 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.582+0000 7f5753fff700 1 --2- 172.21.15.173:0/4057568765 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57540f62a0 0x7f57541064b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:50.576 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.582+0000 7f5753fff700 1 -- 172.21.15.173:0/4057568765 >> 172.21.15.173:0/4057568765 conn(0x7f57540f1aa0 msgr2=0x7f57540f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:50.576 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.582+0000 7f5753fff700 1 -- 172.21.15.173:0/4057568765 shutdown_connections 2024-01-21T19:53:50.576 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:50.582+0000 7f5753fff700 1 -- 172.21.15.173:0/4057568765 wait complete. 2024-01-21T19:53:50.576 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:51.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:50 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:48.982434+0000 mgr.smithi169.espgue (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:51.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:50 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:50.582293+0000 mon.smithi169 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.173:0/4057568765' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:53:51.973 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:53:51.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:53:53.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:52 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:50.982657+0000 mgr.smithi169.espgue (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:53.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.177+0000 7ff122b47700 1 -- 172.21.15.173:0/1920980338 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff11c0f5410 msgr2=0x7ff11c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:53.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.177+0000 7ff122b47700 1 --2- 172.21.15.173:0/1920980338 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff11c0f5410 0x7ff11c0f5830 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7ff110009fb0 tx=0x7ff110009b30).stop 2024-01-21T19:53:53.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.177+0000 7ff122b47700 1 -- 172.21.15.173:0/1920980338 shutdown_connections 2024-01-21T19:53:53.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.177+0000 7ff122b47700 1 --2- 172.21.15.173:0/1920980338 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff11c0f5410 0x7ff11c0f5830 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:53.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.177+0000 7ff122b47700 1 -- 172.21.15.173:0/1920980338 >> 172.21.15.173:0/1920980338 conn(0x7ff11c0f0b70 msgr2=0x7ff11c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:53.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.177+0000 7ff122b47700 1 -- 172.21.15.173:0/1920980338 shutdown_connections 2024-01-21T19:53:53.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.177+0000 7ff122b47700 1 -- 172.21.15.173:0/1920980338 wait complete. 2024-01-21T19:53:53.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.178+0000 7ff122b47700 1 Processor -- start 2024-01-21T19:53:53.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.178+0000 7ff122b47700 1 -- start start 2024-01-21T19:53:53.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.178+0000 7ff122b47700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff11c0f5410 0x7ff11c1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:53.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.178+0000 7ff122b47700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11c060280 con 0x7ff11c0f5410 2024-01-21T19:53:53.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.179+0000 7ff1208e3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff11c0f5410 0x7ff11c1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:53.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.179+0000 7ff1208e3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff11c0f5410 0x7ff11c1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:40364/0 (socket says 172.21.15.173:40364) 2024-01-21T19:53:53.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.179+0000 7ff1208e3700 1 -- 172.21.15.173:0/1012930900 learned_addr learned my addr 172.21.15.173:0/1012930900 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:53:53.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.179+0000 7ff1208e3700 1 -- 172.21.15.173:0/1012930900 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff110005040 con 0x7ff11c0f5410 2024-01-21T19:53:53.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.179+0000 7ff1208e3700 1 --2- 172.21.15.173:0/1012930900 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff11c0f5410 0x7ff11c1093f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7ff110013510 tx=0x7ff11000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:53.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.179+0000 7ff1197fa700 1 -- 172.21.15.173:0/1012930900 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff110009320 con 0x7ff11c0f5410 2024-01-21T19:53:53.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.179+0000 7ff122b47700 1 -- 172.21.15.173:0/1012930900 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff11c060480 con 0x7ff11c0f5410 2024-01-21T19:53:53.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.179+0000 7ff1197fa700 1 -- 172.21.15.173:0/1012930900 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff110013f00 con 0x7ff11c0f5410 2024-01-21T19:53:53.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.179+0000 7ff1197fa700 1 -- 172.21.15.173:0/1012930900 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff110014d90 con 0x7ff11c0f5410 2024-01-21T19:53:53.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.179+0000 7ff122b47700 1 -- 172.21.15.173:0/1012930900 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff11c109b20 con 0x7ff11c0f5410 2024-01-21T19:53:53.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.180+0000 7ff1197fa700 1 -- 172.21.15.173:0/1012930900 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff110014420 con 0x7ff11c0f5410 2024-01-21T19:53:53.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.180+0000 7ff1197fa700 1 --2- 172.21.15.173:0/1012930900 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff10c02ffb0 0x7ff10c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:53.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.180+0000 7ff122b47700 1 -- 172.21.15.173:0/1012930900 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff11c045ad0 con 0x7ff11c0f5410 2024-01-21T19:53:53.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.180+0000 7ff1197fa700 1 -- 172.21.15.173:0/1012930900 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff1100410c0 con 0x7ff11c0f5410 2024-01-21T19:53:53.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.180+0000 7ff11bfff700 1 --2- 172.21.15.173:0/1012930900 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff10c02ffb0 0x7ff10c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:53.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.181+0000 7ff11bfff700 1 --2- 172.21.15.173:0/1012930900 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff10c02ffb0 0x7ff10c032470 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7ff1080096a0 tx=0x7ff108006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:53.177 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.186+0000 7ff1197fa700 1 -- 172.21.15.173:0/1012930900 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff11006b2b0 con 0x7ff11c0f5410 2024-01-21T19:53:53.381 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.389+0000 7ff122b47700 1 -- 172.21.15.173:0/1012930900 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff11c10a340 con 0x7ff11c0f5410 2024-01-21T19:53:53.381 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.390+0000 7ff1197fa700 1 -- 172.21.15.173:0/1012930900 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff11002c090 con 0x7ff11c0f5410 2024-01-21T19:53:53.381 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:53:53.381 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:53:53.383 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.391+0000 7ff122b47700 1 -- 172.21.15.173:0/1012930900 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff10c02ffb0 msgr2=0x7ff10c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:53.383 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.391+0000 7ff122b47700 1 --2- 172.21.15.173:0/1012930900 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff10c02ffb0 0x7ff10c032470 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7ff1080096a0 tx=0x7ff108006b40).stop 2024-01-21T19:53:53.383 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.391+0000 7ff122b47700 1 -- 172.21.15.173:0/1012930900 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff11c0f5410 msgr2=0x7ff11c1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:53.383 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.391+0000 7ff122b47700 1 --2- 172.21.15.173:0/1012930900 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff11c0f5410 0x7ff11c1093f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7ff110013510 tx=0x7ff11000b040).stop 2024-01-21T19:53:53.383 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.391+0000 7ff122b47700 1 -- 172.21.15.173:0/1012930900 shutdown_connections 2024-01-21T19:53:53.383 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.391+0000 7ff122b47700 1 --2- 172.21.15.173:0/1012930900 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff10c02ffb0 0x7ff10c032470 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:53.383 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.391+0000 7ff122b47700 1 --2- 172.21.15.173:0/1012930900 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff11c0f5410 0x7ff11c1093f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:53.383 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.391+0000 7ff122b47700 1 -- 172.21.15.173:0/1012930900 >> 172.21.15.173:0/1012930900 conn(0x7ff11c0f0b70 msgr2=0x7ff11c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:53.384 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.391+0000 7ff122b47700 1 -- 172.21.15.173:0/1012930900 shutdown_connections 2024-01-21T19:53:53.384 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:53.391+0000 7ff122b47700 1 -- 172.21.15.173:0/1012930900 wait complete. 2024-01-21T19:53:53.384 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:53.995 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:53 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:53.390434+0000 mon.smithi169 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.173:0/1012930900' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:53:54.767 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:53:54.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:53:54.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:54 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:52.982904+0000 mgr.smithi169.espgue (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:55.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.949+0000 7fa6b4f69700 1 -- 172.21.15.173:0/3785643492 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6b00f6510 msgr2=0x7fa6b00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:55.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.949+0000 7fa6b4f69700 1 --2- 172.21.15.173:0/3785643492 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6b00f6510 0x7fa6b00f6930 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fa6a4009fb0 tx=0x7fa6a4009b30).stop 2024-01-21T19:53:55.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.949+0000 7fa6b4f69700 1 -- 172.21.15.173:0/3785643492 shutdown_connections 2024-01-21T19:53:55.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.949+0000 7fa6b4f69700 1 --2- 172.21.15.173:0/3785643492 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6b00f6510 0x7fa6b00f6930 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:55.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.949+0000 7fa6b4f69700 1 -- 172.21.15.173:0/3785643492 >> 172.21.15.173:0/3785643492 conn(0x7fa6b00f1c70 msgr2=0x7fa6b00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:55.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.949+0000 7fa6b4f69700 1 -- 172.21.15.173:0/3785643492 shutdown_connections 2024-01-21T19:53:55.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.949+0000 7fa6b4f69700 1 -- 172.21.15.173:0/3785643492 wait complete. 2024-01-21T19:53:55.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.949+0000 7fa6b4f69700 1 Processor -- start 2024-01-21T19:53:55.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.949+0000 7fa6b4f69700 1 -- start start 2024-01-21T19:53:55.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.949+0000 7fa6b4f69700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6b00f6510 0x7fa6b0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:55.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.949+0000 7fa6b4f69700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b0106c20 con 0x7fa6b00f6510 2024-01-21T19:53:55.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.950+0000 7fa6ae59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6b00f6510 0x7fa6b0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:55.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.950+0000 7fa6ae59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6b00f6510 0x7fa6b0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:47822/0 (socket says 172.21.15.173:47822) 2024-01-21T19:53:55.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.950+0000 7fa6ae59c700 1 -- 172.21.15.173:0/379099900 learned_addr learned my addr 172.21.15.173:0/379099900 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:53:55.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.950+0000 7fa6ae59c700 1 -- 172.21.15.173:0/379099900 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a4005040 con 0x7fa6b00f6510 2024-01-21T19:53:55.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.950+0000 7fa6ae59c700 1 --2- 172.21.15.173:0/379099900 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6b00f6510 0x7fa6b0106730 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fa6a4013510 tx=0x7fa6a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:55.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.950+0000 7fa69effd700 1 -- 172.21.15.173:0/379099900 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6a4009320 con 0x7fa6b00f6510 2024-01-21T19:53:55.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.950+0000 7fa6b4f69700 1 -- 172.21.15.173:0/379099900 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b0106e20 con 0x7fa6b00f6510 2024-01-21T19:53:55.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.950+0000 7fa69effd700 1 -- 172.21.15.173:0/379099900 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6a4013f00 con 0x7fa6b00f6510 2024-01-21T19:53:55.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.950+0000 7fa69effd700 1 -- 172.21.15.173:0/379099900 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6a4014d90 con 0x7fa6b00f6510 2024-01-21T19:53:55.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.950+0000 7fa6b4f69700 1 -- 172.21.15.173:0/379099900 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b01072b0 con 0x7fa6b00f6510 2024-01-21T19:53:55.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.951+0000 7fa69effd700 1 -- 172.21.15.173:0/379099900 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6a4014420 con 0x7fa6b00f6510 2024-01-21T19:53:55.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.951+0000 7fa69effd700 1 --2- 172.21.15.173:0/379099900 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa69802ff60 0x7fa698032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:55.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.951+0000 7fa6b4f69700 1 -- 172.21.15.173:0/379099900 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b0045ad0 con 0x7fa6b00f6510 2024-01-21T19:53:55.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.951+0000 7fa69effd700 1 -- 172.21.15.173:0/379099900 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa6a4041070 con 0x7fa6b00f6510 2024-01-21T19:53:55.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.951+0000 7fa6add9b700 1 --2- 172.21.15.173:0/379099900 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa69802ff60 0x7fa698032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:55.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.952+0000 7fa6add9b700 1 --2- 172.21.15.173:0/379099900 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa69802ff60 0x7fa698032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fa6a00096a0 tx=0x7fa6a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:55.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:55.956+0000 7fa69effd700 1 -- 172.21.15.173:0/379099900 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6a40095f0 con 0x7fa6b00f6510 2024-01-21T19:53:56.158 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:56.166+0000 7fa6b4f69700 1 -- 172.21.15.173:0/379099900 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6b00fc050 con 0x7fa6b00f6510 2024-01-21T19:53:56.158 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:56.167+0000 7fa69effd700 1 -- 172.21.15.173:0/379099900 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa6a401db70 con 0x7fa6b00f6510 2024-01-21T19:53:56.159 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:53:56.159 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:53:56.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:56.168+0000 7fa6b4f69700 1 -- 172.21.15.173:0/379099900 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa69802ff60 msgr2=0x7fa698032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:56.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:56.168+0000 7fa6b4f69700 1 --2- 172.21.15.173:0/379099900 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa69802ff60 0x7fa698032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fa6a00096a0 tx=0x7fa6a0006b40).stop 2024-01-21T19:53:56.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:56.168+0000 7fa6b4f69700 1 -- 172.21.15.173:0/379099900 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6b00f6510 msgr2=0x7fa6b0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:56.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:56.168+0000 7fa6b4f69700 1 --2- 172.21.15.173:0/379099900 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6b00f6510 0x7fa6b0106730 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fa6a4013510 tx=0x7fa6a400b040).stop 2024-01-21T19:53:56.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:56.168+0000 7fa6b4f69700 1 -- 172.21.15.173:0/379099900 shutdown_connections 2024-01-21T19:53:56.161 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:56.168+0000 7fa6b4f69700 1 --2- 172.21.15.173:0/379099900 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa69802ff60 0x7fa698032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:56.161 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:56.168+0000 7fa6b4f69700 1 --2- 172.21.15.173:0/379099900 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6b00f6510 0x7fa6b0106730 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:56.161 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:56.168+0000 7fa6b4f69700 1 -- 172.21.15.173:0/379099900 >> 172.21.15.173:0/379099900 conn(0x7fa6b00f1c70 msgr2=0x7fa6b00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:56.161 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:56.168+0000 7fa6b4f69700 1 -- 172.21.15.173:0/379099900 shutdown_connections 2024-01-21T19:53:56.161 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:56.168+0000 7fa6b4f69700 1 -- 172.21.15.173:0/379099900 wait complete. 2024-01-21T19:53:56.161 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:56.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:56 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:54.983157+0000 mgr.smithi169.espgue (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:53:56.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:56 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:55.441500+0000 mon.smithi169 (mon.0) 239 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' 2024-01-21T19:53:56.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:56 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cephadm 2024-01-21T19:53:55.442768+0000 mgr.smithi169.espgue (mgr.14162) 168 : cephadm [INF] Deploying daemon node-exporter.smithi169 on smithi169 2024-01-21T19:53:56.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:56 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:56.167662+0000 mon.smithi169 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.173:0/379099900' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:53:57.552 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:53:57.552 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:53:58.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.753+0000 7f67e486e700 1 -- 172.21.15.173:0/1993939908 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67e00f5410 msgr2=0x7f67e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:58.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.753+0000 7f67e486e700 1 --2- 172.21.15.173:0/1993939908 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67e00f5410 0x7f67e00f5830 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f67d4009fb0 tx=0x7f67d4009b30).stop 2024-01-21T19:53:58.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.753+0000 7f67e486e700 1 -- 172.21.15.173:0/1993939908 shutdown_connections 2024-01-21T19:53:58.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.753+0000 7f67e486e700 1 --2- 172.21.15.173:0/1993939908 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67e00f5410 0x7f67e00f5830 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:58.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.753+0000 7f67e486e700 1 -- 172.21.15.173:0/1993939908 >> 172.21.15.173:0/1993939908 conn(0x7f67e00f0b70 msgr2=0x7f67e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:58.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.753+0000 7f67e486e700 1 -- 172.21.15.173:0/1993939908 shutdown_connections 2024-01-21T19:53:58.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.753+0000 7f67e486e700 1 -- 172.21.15.173:0/1993939908 wait complete. 2024-01-21T19:53:58.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.753+0000 7f67e486e700 1 Processor -- start 2024-01-21T19:53:58.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.753+0000 7f67e486e700 1 -- start start 2024-01-21T19:53:58.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.754+0000 7f67e486e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67e00f5410 0x7f67e010b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:58.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.754+0000 7f67e486e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e01071e0 con 0x7f67e00f5410 2024-01-21T19:53:58.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.754+0000 7f67de59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67e00f5410 0x7f67e010b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:58.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.754+0000 7f67de59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67e00f5410 0x7f67e010b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:47840/0 (socket says 172.21.15.173:47840) 2024-01-21T19:53:58.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.754+0000 7f67de59c700 1 -- 172.21.15.173:0/2240685307 learned_addr learned my addr 172.21.15.173:0/2240685307 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:53:58.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.754+0000 7f67de59c700 1 -- 172.21.15.173:0/2240685307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d4005040 con 0x7f67e00f5410 2024-01-21T19:53:58.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.754+0000 7f67de59c700 1 --2- 172.21.15.173:0/2240685307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67e00f5410 0x7f67e010b7f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f67d4013510 tx=0x7f67d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:58.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.754+0000 7f67ceffd700 1 -- 172.21.15.173:0/2240685307 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67d4009320 con 0x7f67e00f5410 2024-01-21T19:53:58.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.754+0000 7f67ceffd700 1 -- 172.21.15.173:0/2240685307 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67d4013f00 con 0x7f67e00f5410 2024-01-21T19:53:58.748 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.754+0000 7f67e486e700 1 -- 172.21.15.173:0/2240685307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e0060220 con 0x7f67e00f5410 2024-01-21T19:53:58.748 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.754+0000 7f67ceffd700 1 -- 172.21.15.173:0/2240685307 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67d4014d90 con 0x7f67e00f5410 2024-01-21T19:53:58.748 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.754+0000 7f67e486e700 1 -- 172.21.15.173:0/2240685307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e010bce0 con 0x7f67e00f5410 2024-01-21T19:53:58.749 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.755+0000 7f67ceffd700 1 -- 172.21.15.173:0/2240685307 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f67d40095f0 con 0x7f67e00f5410 2024-01-21T19:53:58.749 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.755+0000 7f67e486e700 1 -- 172.21.15.173:0/2240685307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67e0045ad0 con 0x7f67e00f5410 2024-01-21T19:53:58.749 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.755+0000 7f67ceffd700 1 --2- 172.21.15.173:0/2240685307 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f67c802ff60 0x7f67c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:53:58.749 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.755+0000 7f67ceffd700 1 -- 172.21.15.173:0/2240685307 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f67d4041080 con 0x7f67e00f5410 2024-01-21T19:53:58.749 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.755+0000 7f67ddd9b700 1 --2- 172.21.15.173:0/2240685307 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f67c802ff60 0x7f67c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:53:58.749 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.756+0000 7f67ddd9b700 1 --2- 172.21.15.173:0/2240685307 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f67c802ff60 0x7f67c8032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f67d00096a0 tx=0x7f67d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:53:58.751 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.760+0000 7f67ceffd700 1 -- 172.21.15.173:0/2240685307 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f67d4014420 con 0x7f67e00f5410 2024-01-21T19:53:58.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.965+0000 7f67e486e700 1 -- 172.21.15.173:0/2240685307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67e010c4a0 con 0x7f67e00f5410 2024-01-21T19:53:58.958 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.966+0000 7f67ceffd700 1 -- 172.21.15.173:0/2240685307 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f67d401d7f0 con 0x7f67e00f5410 2024-01-21T19:53:58.958 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:53:58.958 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:53:58.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.968+0000 7f67e486e700 1 -- 172.21.15.173:0/2240685307 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f67c802ff60 msgr2=0x7f67c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:58.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.968+0000 7f67e486e700 1 --2- 172.21.15.173:0/2240685307 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f67c802ff60 0x7f67c8032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f67d00096a0 tx=0x7f67d0006b40).stop 2024-01-21T19:53:58.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.968+0000 7f67e486e700 1 -- 172.21.15.173:0/2240685307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67e00f5410 msgr2=0x7f67e010b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:53:58.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.968+0000 7f67e486e700 1 --2- 172.21.15.173:0/2240685307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67e00f5410 0x7f67e010b7f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f67d4013510 tx=0x7f67d400b040).stop 2024-01-21T19:53:58.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.968+0000 7f67e486e700 1 -- 172.21.15.173:0/2240685307 shutdown_connections 2024-01-21T19:53:58.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.968+0000 7f67e486e700 1 --2- 172.21.15.173:0/2240685307 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f67c802ff60 0x7f67c8032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:58.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.968+0000 7f67e486e700 1 --2- 172.21.15.173:0/2240685307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f67e00f5410 0x7f67e010b7f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:53:58.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.968+0000 7f67e486e700 1 -- 172.21.15.173:0/2240685307 >> 172.21.15.173:0/2240685307 conn(0x7f67e00f0b70 msgr2=0x7f67e00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:53:58.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.968+0000 7f67e486e700 1 -- 172.21.15.173:0/2240685307 shutdown_connections 2024-01-21T19:53:58.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:53:58.968+0000 7f67e486e700 1 -- 172.21.15.173:0/2240685307 wait complete. 2024-01-21T19:53:58.961 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:53:59.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:56.983740+0000 mgr.smithi169.espgue (mgr.14162) 169 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:00.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:53:59 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:53:58.966735+0000 mon.smithi169 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.173:0/2240685307' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:00.340 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:00.341 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:54:01.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:00 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:53:58.984068+0000 mgr.smithi169.espgue (mgr.14162) 170 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:01.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.560+0000 7f8551f1b700 1 -- 172.21.15.173:0/3605087321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f854c0f5410 msgr2=0x7f854c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:01.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.560+0000 7f8551f1b700 1 --2- 172.21.15.173:0/3605087321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f854c0f5410 0x7f854c0f5830 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f853c009fb0 tx=0x7f853c009b30).stop 2024-01-21T19:54:01.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.560+0000 7f8551f1b700 1 -- 172.21.15.173:0/3605087321 shutdown_connections 2024-01-21T19:54:01.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.560+0000 7f8551f1b700 1 --2- 172.21.15.173:0/3605087321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f854c0f5410 0x7f854c0f5830 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:01.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.560+0000 7f8551f1b700 1 -- 172.21.15.173:0/3605087321 >> 172.21.15.173:0/3605087321 conn(0x7f854c0f0b70 msgr2=0x7f854c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:01.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.560+0000 7f8551f1b700 1 -- 172.21.15.173:0/3605087321 shutdown_connections 2024-01-21T19:54:01.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.560+0000 7f8551f1b700 1 -- 172.21.15.173:0/3605087321 wait complete. 2024-01-21T19:54:01.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.561+0000 7f8551f1b700 1 Processor -- start 2024-01-21T19:54:01.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.561+0000 7f8551f1b700 1 -- start start 2024-01-21T19:54:01.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.561+0000 7f8551f1b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f854c0f5410 0x7f854c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:01.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.561+0000 7f8551f1b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c060240 con 0x7f854c0f5410 2024-01-21T19:54:01.554 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.561+0000 7f854b7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f854c0f5410 0x7f854c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:01.554 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.561+0000 7f854b7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f854c0f5410 0x7f854c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:47846/0 (socket says 172.21.15.173:47846) 2024-01-21T19:54:01.554 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.561+0000 7f854b7fe700 1 -- 172.21.15.173:0/3512744290 learned_addr learned my addr 172.21.15.173:0/3512744290 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:54:01.554 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.562+0000 7f854b7fe700 1 -- 172.21.15.173:0/3512744290 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c005040 con 0x7f854c0f5410 2024-01-21T19:54:01.554 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.562+0000 7f854b7fe700 1 --2- 172.21.15.173:0/3512744290 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f854c0f5410 0x7f854c109430 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f853c013510 tx=0x7f853c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:01.555 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.562+0000 7f8533fff700 1 -- 172.21.15.173:0/3512744290 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f853c009320 con 0x7f854c0f5410 2024-01-21T19:54:01.555 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.562+0000 7f8551f1b700 1 -- 172.21.15.173:0/3512744290 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f854c060440 con 0x7f854c0f5410 2024-01-21T19:54:01.555 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.562+0000 7f8533fff700 1 -- 172.21.15.173:0/3512744290 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f853c013f00 con 0x7f854c0f5410 2024-01-21T19:54:01.555 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.562+0000 7f8533fff700 1 -- 172.21.15.173:0/3512744290 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f853c014d90 con 0x7f854c0f5410 2024-01-21T19:54:01.555 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.562+0000 7f8551f1b700 1 -- 172.21.15.173:0/3512744290 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f854c109ae0 con 0x7f854c0f5410 2024-01-21T19:54:01.555 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.563+0000 7f8533fff700 1 -- 172.21.15.173:0/3512744290 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f853c0263a0 con 0x7f854c0f5410 2024-01-21T19:54:01.555 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.563+0000 7f8551f1b700 1 -- 172.21.15.173:0/3512744290 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f854c045ad0 con 0x7f854c0f5410 2024-01-21T19:54:01.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.563+0000 7f8533fff700 1 --2- 172.21.15.173:0/3512744290 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f853402ffb0 0x7f8534032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:01.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.563+0000 7f8533fff700 1 -- 172.21.15.173:0/3512744290 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f853c0413c0 con 0x7f854c0f5410 2024-01-21T19:54:01.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.564+0000 7f854affd700 1 --2- 172.21.15.173:0/3512744290 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f853402ffb0 0x7f8534032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:01.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.564+0000 7f854affd700 1 --2- 172.21.15.173:0/3512744290 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f853402ffb0 0x7f8534032470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f85400096a0 tx=0x7f8540006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:01.559 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.568+0000 7f8533fff700 1 -- 172.21.15.173:0/3512744290 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f853c0126f0 con 0x7f854c0f5410 2024-01-21T19:54:01.763 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.771+0000 7f8551f1b700 1 -- 172.21.15.173:0/3512744290 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f854c0fae60 con 0x7f854c0f5410 2024-01-21T19:54:01.763 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.772+0000 7f8533fff700 1 -- 172.21.15.173:0/3512744290 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f853c00f340 con 0x7f854c0f5410 2024-01-21T19:54:01.763 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:54:01.764 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:54:01.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.773+0000 7f8551f1b700 1 -- 172.21.15.173:0/3512744290 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f853402ffb0 msgr2=0x7f8534032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:01.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.773+0000 7f8551f1b700 1 --2- 172.21.15.173:0/3512744290 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f853402ffb0 0x7f8534032470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f85400096a0 tx=0x7f8540006b40).stop 2024-01-21T19:54:01.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.773+0000 7f8551f1b700 1 -- 172.21.15.173:0/3512744290 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f854c0f5410 msgr2=0x7f854c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:01.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.773+0000 7f8551f1b700 1 --2- 172.21.15.173:0/3512744290 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f854c0f5410 0x7f854c109430 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f853c013510 tx=0x7f853c00b040).stop 2024-01-21T19:54:01.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.773+0000 7f8551f1b700 1 -- 172.21.15.173:0/3512744290 shutdown_connections 2024-01-21T19:54:01.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.773+0000 7f8551f1b700 1 --2- 172.21.15.173:0/3512744290 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f853402ffb0 0x7f8534032470 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:01.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.773+0000 7f8551f1b700 1 --2- 172.21.15.173:0/3512744290 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f854c0f5410 0x7f854c109430 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:01.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.773+0000 7f8551f1b700 1 -- 172.21.15.173:0/3512744290 >> 172.21.15.173:0/3512744290 conn(0x7f854c0f0b70 msgr2=0x7f854c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:01.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.773+0000 7f8551f1b700 1 -- 172.21.15.173:0/3512744290 shutdown_connections 2024-01-21T19:54:01.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:01.773+0000 7f8551f1b700 1 -- 172.21.15.173:0/3512744290 wait complete. 2024-01-21T19:54:01.767 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:54:03.157 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:03.157 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:54:03.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:02 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:00.984429+0000 mgr.smithi169.espgue (mgr.14162) 171 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:03.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:02 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:01.772469+0000 mon.smithi169 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.173:0/3512744290' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:04.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.415+0000 7fc182b51700 1 -- 172.21.15.173:0/1138594860 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc17c0f51a0 msgr2=0x7fc17c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:04.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.415+0000 7fc182b51700 1 --2- 172.21.15.173:0/1138594860 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc17c0f51a0 0x7fc17c0f55c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fc178009fb0 tx=0x7fc178009b30).stop 2024-01-21T19:54:04.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.415+0000 7fc182b51700 1 -- 172.21.15.173:0/1138594860 shutdown_connections 2024-01-21T19:54:04.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.415+0000 7fc182b51700 1 --2- 172.21.15.173:0/1138594860 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc17c0f51a0 0x7fc17c0f55c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:04.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.415+0000 7fc182b51700 1 -- 172.21.15.173:0/1138594860 >> 172.21.15.173:0/1138594860 conn(0x7fc17c0f09a0 msgr2=0x7fc17c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:04.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.415+0000 7fc182b51700 1 -- 172.21.15.173:0/1138594860 shutdown_connections 2024-01-21T19:54:04.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.415+0000 7fc182b51700 1 -- 172.21.15.173:0/1138594860 wait complete. 2024-01-21T19:54:04.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.415+0000 7fc182b51700 1 Processor -- start 2024-01-21T19:54:04.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.415+0000 7fc182b51700 1 -- start start 2024-01-21T19:54:04.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.415+0000 7fc182b51700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc17c0f51a0 0x7fc17c10b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:04.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.415+0000 7fc182b51700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c106fc0 con 0x7fc17c0f51a0 2024-01-21T19:54:04.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.416+0000 7fc181b4f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc17c0f51a0 0x7fc17c10b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:04.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.416+0000 7fc181b4f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc17c0f51a0 0x7fc17c10b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:57820/0 (socket says 172.21.15.173:57820) 2024-01-21T19:54:04.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.416+0000 7fc181b4f700 1 -- 172.21.15.173:0/3234704843 learned_addr learned my addr 172.21.15.173:0/3234704843 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:54:04.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.416+0000 7fc181b4f700 1 -- 172.21.15.173:0/3234704843 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc178005040 con 0x7fc17c0f51a0 2024-01-21T19:54:04.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.416+0000 7fc181b4f700 1 --2- 172.21.15.173:0/3234704843 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc17c0f51a0 0x7fc17c10b5d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fc178013510 tx=0x7fc17800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:04.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.416+0000 7fc1727fc700 1 -- 172.21.15.173:0/3234704843 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc178009320 con 0x7fc17c0f51a0 2024-01-21T19:54:04.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.416+0000 7fc1727fc700 1 -- 172.21.15.173:0/3234704843 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc178013f00 con 0x7fc17c0f51a0 2024-01-21T19:54:04.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.416+0000 7fc182b51700 1 -- 172.21.15.173:0/3234704843 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17c060220 con 0x7fc17c0f51a0 2024-01-21T19:54:04.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.416+0000 7fc1727fc700 1 -- 172.21.15.173:0/3234704843 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc178014dc0 con 0x7fc17c0f51a0 2024-01-21T19:54:04.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.416+0000 7fc182b51700 1 -- 172.21.15.173:0/3234704843 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc17c10bac0 con 0x7fc17c0f51a0 2024-01-21T19:54:04.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.417+0000 7fc1727fc700 1 -- 172.21.15.173:0/3234704843 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc17801d3d0 con 0x7fc17c0f51a0 2024-01-21T19:54:04.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.417+0000 7fc1727fc700 1 --2- 172.21.15.173:0/3234704843 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc16802ff60 0x7fc168032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:04.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.417+0000 7fc1727fc700 1 -- 172.21.15.173:0/3234704843 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc178040ff0 con 0x7fc17c0f51a0 2024-01-21T19:54:04.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.417+0000 7fc182b51700 1 -- 172.21.15.173:0/3234704843 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc17c059100 con 0x7fc17c0f51a0 2024-01-21T19:54:04.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.417+0000 7fc18134e700 1 --2- 172.21.15.173:0/3234704843 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc16802ff60 0x7fc168032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:04.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.418+0000 7fc18134e700 1 --2- 172.21.15.173:0/3234704843 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc16802ff60 0x7fc168032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fc16c0096a0 tx=0x7fc16c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:04.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.422+0000 7fc1727fc700 1 -- 172.21.15.173:0/3234704843 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc178012900 con 0x7fc17c0f51a0 2024-01-21T19:54:04.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.629+0000 7fc182b51700 1 -- 172.21.15.173:0/3234704843 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc17c045ad0 con 0x7fc17c0f51a0 2024-01-21T19:54:04.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.630+0000 7fc1727fc700 1 -- 172.21.15.173:0/3234704843 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc17802b020 con 0x7fc17c0f51a0 2024-01-21T19:54:04.622 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:54:04.622 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:54:04.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.632+0000 7fc182b51700 1 -- 172.21.15.173:0/3234704843 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc16802ff60 msgr2=0x7fc168032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:04.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.632+0000 7fc182b51700 1 --2- 172.21.15.173:0/3234704843 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc16802ff60 0x7fc168032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fc16c0096a0 tx=0x7fc16c006b40).stop 2024-01-21T19:54:04.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.632+0000 7fc182b51700 1 -- 172.21.15.173:0/3234704843 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc17c0f51a0 msgr2=0x7fc17c10b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:04.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.632+0000 7fc182b51700 1 --2- 172.21.15.173:0/3234704843 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc17c0f51a0 0x7fc17c10b5d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fc178013510 tx=0x7fc17800b040).stop 2024-01-21T19:54:04.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.632+0000 7fc182b51700 1 -- 172.21.15.173:0/3234704843 shutdown_connections 2024-01-21T19:54:04.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.632+0000 7fc182b51700 1 --2- 172.21.15.173:0/3234704843 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc16802ff60 0x7fc168032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:04.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.632+0000 7fc182b51700 1 --2- 172.21.15.173:0/3234704843 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc17c0f51a0 0x7fc17c10b5d0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:04.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.632+0000 7fc182b51700 1 -- 172.21.15.173:0/3234704843 >> 172.21.15.173:0/3234704843 conn(0x7fc17c0f09a0 msgr2=0x7fc17c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:04.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.632+0000 7fc182b51700 1 -- 172.21.15.173:0/3234704843 shutdown_connections 2024-01-21T19:54:04.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:04.632+0000 7fc182b51700 1 -- 172.21.15.173:0/3234704843 wait complete. 2024-01-21T19:54:04.624 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:54:05.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:04 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:02.984796+0000 mgr.smithi169.espgue (mgr.14162) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:05.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:04 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:04.630662+0000 mon.smithi169 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.173:0/3234704843' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:06.027 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:06.027 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:54:07.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:06 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:04.985213+0000 mgr.smithi169.espgue (mgr.14162) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:07.265 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.272+0000 7ff2d898c700 1 -- 172.21.15.173:0/60337487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2d40f5410 msgr2=0x7ff2d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:07.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.272+0000 7ff2d898c700 1 --2- 172.21.15.173:0/60337487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2d40f5410 0x7ff2d40f5830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7ff2bc009fb0 tx=0x7ff2bc009b30).stop 2024-01-21T19:54:07.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.272+0000 7ff2d898c700 1 -- 172.21.15.173:0/60337487 shutdown_connections 2024-01-21T19:54:07.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.272+0000 7ff2d898c700 1 --2- 172.21.15.173:0/60337487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2d40f5410 0x7ff2d40f5830 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:07.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.272+0000 7ff2d898c700 1 -- 172.21.15.173:0/60337487 >> 172.21.15.173:0/60337487 conn(0x7ff2d40f0b70 msgr2=0x7ff2d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:07.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.272+0000 7ff2d898c700 1 -- 172.21.15.173:0/60337487 shutdown_connections 2024-01-21T19:54:07.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.272+0000 7ff2d898c700 1 -- 172.21.15.173:0/60337487 wait complete. 2024-01-21T19:54:07.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.273+0000 7ff2d898c700 1 Processor -- start 2024-01-21T19:54:07.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.273+0000 7ff2d898c700 1 -- start start 2024-01-21T19:54:07.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.273+0000 7ff2d898c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2d40f5410 0x7ff2d4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:07.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.273+0000 7ff2d898c700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d4105b20 con 0x7ff2d40f5410 2024-01-21T19:54:07.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.273+0000 7ff2d259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2d40f5410 0x7ff2d4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:07.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.273+0000 7ff2d259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2d40f5410 0x7ff2d4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:57826/0 (socket says 172.21.15.173:57826) 2024-01-21T19:54:07.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.273+0000 7ff2d259c700 1 -- 172.21.15.173:0/3804142088 learned_addr learned my addr 172.21.15.173:0/3804142088 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:54:07.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.273+0000 7ff2d259c700 1 -- 172.21.15.173:0/3804142088 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2bc005040 con 0x7ff2d40f5410 2024-01-21T19:54:07.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.274+0000 7ff2d259c700 1 --2- 172.21.15.173:0/3804142088 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2d40f5410 0x7ff2d4105630 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7ff2bc013510 tx=0x7ff2bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:07.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.274+0000 7ff2caffd700 1 -- 172.21.15.173:0/3804142088 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2bc009320 con 0x7ff2d40f5410 2024-01-21T19:54:07.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.274+0000 7ff2caffd700 1 -- 172.21.15.173:0/3804142088 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2bc013f00 con 0x7ff2d40f5410 2024-01-21T19:54:07.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.274+0000 7ff2d898c700 1 -- 172.21.15.173:0/3804142088 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2d4105d20 con 0x7ff2d40f5410 2024-01-21T19:54:07.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.274+0000 7ff2caffd700 1 -- 172.21.15.173:0/3804142088 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2bc014d90 con 0x7ff2d40f5410 2024-01-21T19:54:07.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.274+0000 7ff2d898c700 1 -- 172.21.15.173:0/3804142088 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2d41061b0 con 0x7ff2d40f5410 2024-01-21T19:54:07.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.275+0000 7ff2caffd700 1 -- 172.21.15.173:0/3804142088 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2bc0095f0 con 0x7ff2d40f5410 2024-01-21T19:54:07.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.275+0000 7ff2d898c700 1 -- 172.21.15.173:0/3804142088 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2d4045ad0 con 0x7ff2d40f5410 2024-01-21T19:54:07.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.275+0000 7ff2caffd700 1 --2- 172.21.15.173:0/3804142088 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff2c002ff60 0x7ff2c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:07.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.275+0000 7ff2caffd700 1 -- 172.21.15.173:0/3804142088 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff2bc041000 con 0x7ff2d40f5410 2024-01-21T19:54:07.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.275+0000 7ff2d1d9b700 1 --2- 172.21.15.173:0/3804142088 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff2c002ff60 0x7ff2c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:07.269 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.275+0000 7ff2d1d9b700 1 --2- 172.21.15.173:0/3804142088 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff2c002ff60 0x7ff2c0032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7ff2c40096a0 tx=0x7ff2c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:07.271 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.279+0000 7ff2caffd700 1 -- 172.21.15.173:0/3804142088 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2bc01d7f0 con 0x7ff2d40f5410 2024-01-21T19:54:07.478 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.486+0000 7ff2d898c700 1 -- 172.21.15.173:0/3804142088 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2d40fae60 con 0x7ff2d40f5410 2024-01-21T19:54:07.478 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.487+0000 7ff2caffd700 1 -- 172.21.15.173:0/3804142088 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff2bc010230 con 0x7ff2d40f5410 2024-01-21T19:54:07.479 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:54:07.479 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:54:07.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.488+0000 7ff2d898c700 1 -- 172.21.15.173:0/3804142088 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff2c002ff60 msgr2=0x7ff2c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:07.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.488+0000 7ff2d898c700 1 --2- 172.21.15.173:0/3804142088 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff2c002ff60 0x7ff2c0032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7ff2c40096a0 tx=0x7ff2c4006b40).stop 2024-01-21T19:54:07.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.488+0000 7ff2d898c700 1 -- 172.21.15.173:0/3804142088 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2d40f5410 msgr2=0x7ff2d4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:07.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.488+0000 7ff2d898c700 1 --2- 172.21.15.173:0/3804142088 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2d40f5410 0x7ff2d4105630 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7ff2bc013510 tx=0x7ff2bc00b040).stop 2024-01-21T19:54:07.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.489+0000 7ff2d898c700 1 -- 172.21.15.173:0/3804142088 shutdown_connections 2024-01-21T19:54:07.481 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.489+0000 7ff2d898c700 1 --2- 172.21.15.173:0/3804142088 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ff2c002ff60 0x7ff2c0032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:07.481 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.489+0000 7ff2d898c700 1 --2- 172.21.15.173:0/3804142088 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff2d40f5410 0x7ff2d4105630 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:07.481 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.489+0000 7ff2d898c700 1 -- 172.21.15.173:0/3804142088 >> 172.21.15.173:0/3804142088 conn(0x7ff2d40f0b70 msgr2=0x7ff2d40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:07.481 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.489+0000 7ff2d898c700 1 -- 172.21.15.173:0/3804142088 shutdown_connections 2024-01-21T19:54:07.481 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:07.489+0000 7ff2d898c700 1 -- 172.21.15.173:0/3804142088 wait complete. 2024-01-21T19:54:07.481 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:54:08.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:07 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:07.487531+0000 mon.smithi169 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.173:0/3804142088' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:08.871 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:08.871 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:54:09.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:08 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:06.985511+0000 mgr.smithi169.espgue (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:10.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.070+0000 7fbebb59e700 1 -- 172.21.15.173:0/3362919384 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbebc0f51a0 msgr2=0x7fbebc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:10.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.070+0000 7fbebb59e700 1 --2- 172.21.15.173:0/3362919384 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbebc0f51a0 0x7fbebc0f55c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fbeac009fb0 tx=0x7fbeac009b30).stop 2024-01-21T19:54:10.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.071+0000 7fbebb59e700 1 -- 172.21.15.173:0/3362919384 shutdown_connections 2024-01-21T19:54:10.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.071+0000 7fbebb59e700 1 --2- 172.21.15.173:0/3362919384 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbebc0f51a0 0x7fbebc0f55c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:10.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.071+0000 7fbebb59e700 1 -- 172.21.15.173:0/3362919384 >> 172.21.15.173:0/3362919384 conn(0x7fbebc0f09a0 msgr2=0x7fbebc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:10.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.071+0000 7fbebb59e700 1 -- 172.21.15.173:0/3362919384 shutdown_connections 2024-01-21T19:54:10.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.071+0000 7fbebb59e700 1 -- 172.21.15.173:0/3362919384 wait complete. 2024-01-21T19:54:10.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.071+0000 7fbebb59e700 1 Processor -- start 2024-01-21T19:54:10.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.071+0000 7fbebb59e700 1 -- start start 2024-01-21T19:54:10.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.071+0000 7fbebb59e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbebc0f51a0 0x7fbebc1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:10.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.071+0000 7fbebb59e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc060220 con 0x7fbebc0f51a0 2024-01-21T19:54:10.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.072+0000 7fbeba59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbebc0f51a0 0x7fbebc1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:10.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.072+0000 7fbeba59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbebc0f51a0 0x7fbebc1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:57836/0 (socket says 172.21.15.173:57836) 2024-01-21T19:54:10.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.072+0000 7fbeba59c700 1 -- 172.21.15.173:0/851319861 learned_addr learned my addr 172.21.15.173:0/851319861 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:54:10.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.072+0000 7fbeba59c700 1 -- 172.21.15.173:0/851319861 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeac005040 con 0x7fbebc0f51a0 2024-01-21T19:54:10.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.072+0000 7fbeba59c700 1 --2- 172.21.15.173:0/851319861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbebc0f51a0 0x7fbebc1091d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fbeac013510 tx=0x7fbeac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:10.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.072+0000 7fbeaaffd700 1 -- 172.21.15.173:0/851319861 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbeac009320 con 0x7fbebc0f51a0 2024-01-21T19:54:10.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.072+0000 7fbebb59e700 1 -- 172.21.15.173:0/851319861 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbebc060420 con 0x7fbebc0f51a0 2024-01-21T19:54:10.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.072+0000 7fbeaaffd700 1 -- 172.21.15.173:0/851319861 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbeac013f00 con 0x7fbebc0f51a0 2024-01-21T19:54:10.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.072+0000 7fbeaaffd700 1 -- 172.21.15.173:0/851319861 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbeac014dc0 con 0x7fbebc0f51a0 2024-01-21T19:54:10.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.072+0000 7fbebb59e700 1 -- 172.21.15.173:0/851319861 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbebc109880 con 0x7fbebc0f51a0 2024-01-21T19:54:10.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.073+0000 7fbeaaffd700 1 -- 172.21.15.173:0/851319861 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbeac01d3d0 con 0x7fbebc0f51a0 2024-01-21T19:54:10.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.073+0000 7fbeaaffd700 1 --2- 172.21.15.173:0/851319861 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fbea402ff60 0x7fbea4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:10.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.073+0000 7fbeaaffd700 1 -- 172.21.15.173:0/851319861 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbeac0413a0 con 0x7fbebc0f51a0 2024-01-21T19:54:10.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.073+0000 7fbebb59e700 1 -- 172.21.15.173:0/851319861 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbebc059100 con 0x7fbebc0f51a0 2024-01-21T19:54:10.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.073+0000 7fbeb9d9b700 1 --2- 172.21.15.173:0/851319861 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fbea402ff60 0x7fbea4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:10.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.074+0000 7fbeb9d9b700 1 --2- 172.21.15.173:0/851319861 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fbea402ff60 0x7fbea4032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fbeb00096a0 tx=0x7fbeb0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:10.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.078+0000 7fbeaaffd700 1 -- 172.21.15.173:0/851319861 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbeac012900 con 0x7fbebc0f51a0 2024-01-21T19:54:10.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.283+0000 7fbebb59e700 1 -- 172.21.15.173:0/851319861 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbebc0fabe0 con 0x7fbebc0f51a0 2024-01-21T19:54:10.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.284+0000 7fbeaaffd700 1 -- 172.21.15.173:0/851319861 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbeac02b030 con 0x7fbebc0f51a0 2024-01-21T19:54:10.275 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:54:10.275 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:54:10.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.285+0000 7fbebb59e700 1 -- 172.21.15.173:0/851319861 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fbea402ff60 msgr2=0x7fbea4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:10.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.285+0000 7fbebb59e700 1 --2- 172.21.15.173:0/851319861 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fbea402ff60 0x7fbea4032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fbeb00096a0 tx=0x7fbeb0006b40).stop 2024-01-21T19:54:10.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.285+0000 7fbebb59e700 1 -- 172.21.15.173:0/851319861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbebc0f51a0 msgr2=0x7fbebc1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:10.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.285+0000 7fbebb59e700 1 --2- 172.21.15.173:0/851319861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbebc0f51a0 0x7fbebc1091d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fbeac013510 tx=0x7fbeac00b040).stop 2024-01-21T19:54:10.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.285+0000 7fbebb59e700 1 -- 172.21.15.173:0/851319861 shutdown_connections 2024-01-21T19:54:10.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.285+0000 7fbebb59e700 1 --2- 172.21.15.173:0/851319861 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fbea402ff60 0x7fbea4032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:10.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.285+0000 7fbebb59e700 1 --2- 172.21.15.173:0/851319861 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbebc0f51a0 0x7fbebc1091d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:10.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.285+0000 7fbebb59e700 1 -- 172.21.15.173:0/851319861 >> 172.21.15.173:0/851319861 conn(0x7fbebc0f09a0 msgr2=0x7fbebc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:10.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.285+0000 7fbebb59e700 1 -- 172.21.15.173:0/851319861 shutdown_connections 2024-01-21T19:54:10.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:10.285+0000 7fbebb59e700 1 -- 172.21.15.173:0/851319861 wait complete. 2024-01-21T19:54:10.278 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:54:11.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:10 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:08.985822+0000 mgr.smithi169.espgue (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:11.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:10 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:10.284476+0000 mon.smithi169 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.173:0/851319861' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:11.671 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:11.672 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:54:12.888 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.896+0000 7fc08e59a700 1 -- 172.21.15.173:0/3958454479 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0880f5410 msgr2=0x7fc0880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:12.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.896+0000 7fc08e59a700 1 --2- 172.21.15.173:0/3958454479 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0880f5410 0x7fc0880f5830 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fc078009fb0 tx=0x7fc078009b30).stop 2024-01-21T19:54:12.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.896+0000 7fc08e59a700 1 -- 172.21.15.173:0/3958454479 shutdown_connections 2024-01-21T19:54:12.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.896+0000 7fc08e59a700 1 --2- 172.21.15.173:0/3958454479 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0880f5410 0x7fc0880f5830 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:12.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.896+0000 7fc08e59a700 1 -- 172.21.15.173:0/3958454479 >> 172.21.15.173:0/3958454479 conn(0x7fc0880f0b70 msgr2=0x7fc0880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:12.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.897+0000 7fc08e59a700 1 -- 172.21.15.173:0/3958454479 shutdown_connections 2024-01-21T19:54:12.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.897+0000 7fc08e59a700 1 -- 172.21.15.173:0/3958454479 wait complete. 2024-01-21T19:54:12.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.897+0000 7fc08e59a700 1 Processor -- start 2024-01-21T19:54:12.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.897+0000 7fc08e59a700 1 -- start start 2024-01-21T19:54:12.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.898+0000 7fc08e59a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0880f5410 0x7fc088105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:12.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.898+0000 7fc08e59a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc088105b20 con 0x7fc0880f5410 2024-01-21T19:54:12.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.898+0000 7fc087fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0880f5410 0x7fc088105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:12.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.898+0000 7fc087fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0880f5410 0x7fc088105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:57854/0 (socket says 172.21.15.173:57854) 2024-01-21T19:54:12.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.898+0000 7fc087fff700 1 -- 172.21.15.173:0/2873801305 learned_addr learned my addr 172.21.15.173:0/2873801305 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:54:12.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.898+0000 7fc087fff700 1 -- 172.21.15.173:0/2873801305 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc078005040 con 0x7fc0880f5410 2024-01-21T19:54:12.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.898+0000 7fc087fff700 1 --2- 172.21.15.173:0/2873801305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0880f5410 0x7fc088105630 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fc078014000 tx=0x7fc07800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:12.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.898+0000 7fc084ff9700 1 -- 172.21.15.173:0/2873801305 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0780095f0 con 0x7fc0880f5410 2024-01-21T19:54:12.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.898+0000 7fc084ff9700 1 -- 172.21.15.173:0/2873801305 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc078007e70 con 0x7fc0880f5410 2024-01-21T19:54:12.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.899+0000 7fc08e59a700 1 -- 172.21.15.173:0/2873801305 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc088105d20 con 0x7fc0880f5410 2024-01-21T19:54:12.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.899+0000 7fc084ff9700 1 -- 172.21.15.173:0/2873801305 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc078014ea0 con 0x7fc0880f5410 2024-01-21T19:54:12.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.899+0000 7fc08e59a700 1 -- 172.21.15.173:0/2873801305 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0881061b0 con 0x7fc0880f5410 2024-01-21T19:54:12.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.899+0000 7fc084ff9700 1 -- 172.21.15.173:0/2873801305 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc078027070 con 0x7fc0880f5410 2024-01-21T19:54:12.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.899+0000 7fc08e59a700 1 -- 172.21.15.173:0/2873801305 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc088045ad0 con 0x7fc0880f5410 2024-01-21T19:54:12.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.900+0000 7fc084ff9700 1 --2- 172.21.15.173:0/2873801305 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc07002ff60 0x7fc070032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:12.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.900+0000 7fc084ff9700 1 -- 172.21.15.173:0/2873801305 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc078042710 con 0x7fc0880f5410 2024-01-21T19:54:12.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.900+0000 7fc0877fe700 1 --2- 172.21.15.173:0/2873801305 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc07002ff60 0x7fc070032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:12.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.900+0000 7fc0877fe700 1 --2- 172.21.15.173:0/2873801305 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc07002ff60 0x7fc070032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fc07c0096a0 tx=0x7fc07c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:12.896 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:12.904+0000 7fc084ff9700 1 -- 172.21.15.173:0/2873801305 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc0780274e0 con 0x7fc0880f5410 2024-01-21T19:54:13.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:13.109+0000 7fc08e59a700 1 -- 172.21.15.173:0/2873801305 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0880fae60 con 0x7fc0880f5410 2024-01-21T19:54:13.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:13.109+0000 7fc084ff9700 1 -- 172.21.15.173:0/2873801305 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc078027300 con 0x7fc0880f5410 2024-01-21T19:54:13.101 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:54:13.101 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:54:13.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:13.111+0000 7fc08e59a700 1 -- 172.21.15.173:0/2873801305 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc07002ff60 msgr2=0x7fc070032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:13.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:13.111+0000 7fc08e59a700 1 --2- 172.21.15.173:0/2873801305 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc07002ff60 0x7fc070032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fc07c0096a0 tx=0x7fc07c006b40).stop 2024-01-21T19:54:13.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:13.111+0000 7fc08e59a700 1 -- 172.21.15.173:0/2873801305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0880f5410 msgr2=0x7fc088105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:13.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:13.111+0000 7fc08e59a700 1 --2- 172.21.15.173:0/2873801305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0880f5410 0x7fc088105630 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fc078014000 tx=0x7fc07800b040).stop 2024-01-21T19:54:13.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:13.111+0000 7fc08e59a700 1 -- 172.21.15.173:0/2873801305 shutdown_connections 2024-01-21T19:54:13.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:13.111+0000 7fc08e59a700 1 --2- 172.21.15.173:0/2873801305 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc07002ff60 0x7fc070032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:13.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:13.111+0000 7fc08e59a700 1 --2- 172.21.15.173:0/2873801305 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc0880f5410 0x7fc088105630 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:13.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:13.111+0000 7fc08e59a700 1 -- 172.21.15.173:0/2873801305 >> 172.21.15.173:0/2873801305 conn(0x7fc0880f0b70 msgr2=0x7fc0880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:13.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:13.111+0000 7fc08e59a700 1 -- 172.21.15.173:0/2873801305 shutdown_connections 2024-01-21T19:54:13.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:13.111+0000 7fc08e59a700 1 -- 172.21.15.173:0/2873801305 wait complete. 2024-01-21T19:54:13.103 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:54:13.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:12 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:10.986184+0000 mgr.smithi169.espgue (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:14.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:13 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:13.110036+0000 mon.smithi169 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.173:0/2873801305' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:14.496 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:14.499 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:54:15.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:14 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:12.986530+0000 mgr.smithi169.espgue (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:15.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.744+0000 7f69dd8b7700 1 -- 172.21.15.173:0/3574474572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69d80f5410 msgr2=0x7f69d80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:15.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.744+0000 7f69dd8b7700 1 --2- 172.21.15.173:0/3574474572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69d80f5410 0x7f69d80f5830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f69cc009fb0 tx=0x7f69cc009b30).stop 2024-01-21T19:54:15.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.744+0000 7f69dd8b7700 1 -- 172.21.15.173:0/3574474572 shutdown_connections 2024-01-21T19:54:15.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.744+0000 7f69dd8b7700 1 --2- 172.21.15.173:0/3574474572 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69d80f5410 0x7f69d80f5830 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:15.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.744+0000 7f69dd8b7700 1 -- 172.21.15.173:0/3574474572 >> 172.21.15.173:0/3574474572 conn(0x7f69d80f0b70 msgr2=0x7f69d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:15.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.744+0000 7f69dd8b7700 1 -- 172.21.15.173:0/3574474572 shutdown_connections 2024-01-21T19:54:15.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.744+0000 7f69dd8b7700 1 -- 172.21.15.173:0/3574474572 wait complete. 2024-01-21T19:54:15.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.745+0000 7f69dd8b7700 1 Processor -- start 2024-01-21T19:54:15.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.745+0000 7f69dd8b7700 1 -- start start 2024-01-21T19:54:15.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.745+0000 7f69dd8b7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69d80f5410 0x7f69d8106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:15.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.745+0000 7f69dd8b7700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d8106bf0 con 0x7f69d80f5410 2024-01-21T19:54:15.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.745+0000 7f69d6ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69d80f5410 0x7f69d8106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:15.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.745+0000 7f69d6ffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69d80f5410 0x7f69d8106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:55022/0 (socket says 172.21.15.173:55022) 2024-01-21T19:54:15.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.745+0000 7f69d6ffd700 1 -- 172.21.15.173:0/735201661 learned_addr learned my addr 172.21.15.173:0/735201661 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:54:15.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.745+0000 7f69d6ffd700 1 -- 172.21.15.173:0/735201661 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69cc005040 con 0x7f69d80f5410 2024-01-21T19:54:15.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.745+0000 7f69d6ffd700 1 --2- 172.21.15.173:0/735201661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69d80f5410 0x7f69d8106700 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f69cc013510 tx=0x7f69cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:15.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.745+0000 7f69bf7fe700 1 -- 172.21.15.173:0/735201661 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69cc009320 con 0x7f69d80f5410 2024-01-21T19:54:15.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.746+0000 7f69dd8b7700 1 -- 172.21.15.173:0/735201661 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69d8106df0 con 0x7f69d80f5410 2024-01-21T19:54:15.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.746+0000 7f69bf7fe700 1 -- 172.21.15.173:0/735201661 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69cc013f00 con 0x7f69d80f5410 2024-01-21T19:54:15.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.746+0000 7f69bf7fe700 1 -- 172.21.15.173:0/735201661 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69cc014d90 con 0x7f69d80f5410 2024-01-21T19:54:15.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.746+0000 7f69dd8b7700 1 -- 172.21.15.173:0/735201661 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69d8107280 con 0x7f69d80f5410 2024-01-21T19:54:15.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.746+0000 7f69bf7fe700 1 -- 172.21.15.173:0/735201661 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69cc0095f0 con 0x7f69d80f5410 2024-01-21T19:54:15.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.746+0000 7f69dd8b7700 1 -- 172.21.15.173:0/735201661 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69d8045ad0 con 0x7f69d80f5410 2024-01-21T19:54:15.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.746+0000 7f69bf7fe700 1 --2- 172.21.15.173:0/735201661 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f69c002ff60 0x7f69c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:15.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.747+0000 7f69bf7fe700 1 -- 172.21.15.173:0/735201661 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f69cc041000 con 0x7f69d80f5410 2024-01-21T19:54:15.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.747+0000 7f69d67fc700 1 --2- 172.21.15.173:0/735201661 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f69c002ff60 0x7f69c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:15.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.747+0000 7f69d67fc700 1 --2- 172.21.15.173:0/735201661 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f69c002ff60 0x7f69c0032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f69c80096a0 tx=0x7f69c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:15.742 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.751+0000 7f69bf7fe700 1 -- 172.21.15.173:0/735201661 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69cc014420 con 0x7f69d80f5410 2024-01-21T19:54:15.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.954+0000 7f69dd8b7700 1 -- 172.21.15.173:0/735201661 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69d80fae60 con 0x7f69d80f5410 2024-01-21T19:54:15.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.955+0000 7f69bf7fe700 1 -- 172.21.15.173:0/735201661 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f69cc010230 con 0x7f69d80f5410 2024-01-21T19:54:15.946 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:54:15.947 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:54:15.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.956+0000 7f69dd8b7700 1 -- 172.21.15.173:0/735201661 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f69c002ff60 msgr2=0x7f69c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:15.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.956+0000 7f69dd8b7700 1 --2- 172.21.15.173:0/735201661 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f69c002ff60 0x7f69c0032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f69c80096a0 tx=0x7f69c8006b40).stop 2024-01-21T19:54:15.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.956+0000 7f69dd8b7700 1 -- 172.21.15.173:0/735201661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69d80f5410 msgr2=0x7f69d8106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:15.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.956+0000 7f69dd8b7700 1 --2- 172.21.15.173:0/735201661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69d80f5410 0x7f69d8106700 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f69cc013510 tx=0x7f69cc00b040).stop 2024-01-21T19:54:15.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.956+0000 7f69dd8b7700 1 -- 172.21.15.173:0/735201661 shutdown_connections 2024-01-21T19:54:15.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.956+0000 7f69dd8b7700 1 --2- 172.21.15.173:0/735201661 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f69c002ff60 0x7f69c0032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:15.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.956+0000 7f69dd8b7700 1 --2- 172.21.15.173:0/735201661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69d80f5410 0x7f69d8106700 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:15.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.956+0000 7f69dd8b7700 1 -- 172.21.15.173:0/735201661 >> 172.21.15.173:0/735201661 conn(0x7f69d80f0b70 msgr2=0x7f69d80f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:15.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.957+0000 7f69dd8b7700 1 -- 172.21.15.173:0/735201661 shutdown_connections 2024-01-21T19:54:15.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:15.957+0000 7f69dd8b7700 1 -- 172.21.15.173:0/735201661 wait complete. 2024-01-21T19:54:15.949 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:54:17.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:16 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:14.986890+0000 mgr.smithi169.espgue (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:17.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:16 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:15.955675+0000 mon.smithi169 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.173:0/735201661' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:17.323 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:17.323 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:54:18.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.547+0000 7f80f0896700 1 -- 172.21.15.173:0/1535273943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80ec060180 msgr2=0x7f80ec0f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:18.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.547+0000 7f80f0896700 1 --2- 172.21.15.173:0/1535273943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80ec060180 0x7f80ec0f8f80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f80dc009fb0 tx=0x7f80dc009b30).stop 2024-01-21T19:54:18.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.547+0000 7f80f0896700 1 -- 172.21.15.173:0/1535273943 shutdown_connections 2024-01-21T19:54:18.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.547+0000 7f80f0896700 1 --2- 172.21.15.173:0/1535273943 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80ec060180 0x7f80ec0f8f80 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:18.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.547+0000 7f80f0896700 1 -- 172.21.15.173:0/1535273943 >> 172.21.15.173:0/1535273943 conn(0x7f80ec0f09a0 msgr2=0x7f80ec0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:18.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.547+0000 7f80f0896700 1 -- 172.21.15.173:0/1535273943 shutdown_connections 2024-01-21T19:54:18.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.547+0000 7f80f0896700 1 -- 172.21.15.173:0/1535273943 wait complete. 2024-01-21T19:54:18.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.548+0000 7f80f0896700 1 Processor -- start 2024-01-21T19:54:18.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.548+0000 7f80f0896700 1 -- start start 2024-01-21T19:54:18.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.548+0000 7f80f0896700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80ec060180 0x7f80ec0f6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:18.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.548+0000 7f80f0896700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec0f4e80 con 0x7f80ec060180 2024-01-21T19:54:18.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.549+0000 7f80eb7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80ec060180 0x7f80ec0f6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:18.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.549+0000 7f80eb7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80ec060180 0x7f80ec0f6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:55044/0 (socket says 172.21.15.173:55044) 2024-01-21T19:54:18.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.549+0000 7f80eb7fe700 1 -- 172.21.15.173:0/3917561722 learned_addr learned my addr 172.21.15.173:0/3917561722 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:54:18.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.549+0000 7f80eb7fe700 1 -- 172.21.15.173:0/3917561722 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80dc005040 con 0x7f80ec060180 2024-01-21T19:54:18.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.549+0000 7f80eb7fe700 1 --2- 172.21.15.173:0/3917561722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80ec060180 0x7f80ec0f6a00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f80dc013510 tx=0x7f80dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:18.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.549+0000 7f80d3fff700 1 -- 172.21.15.173:0/3917561722 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80dc009320 con 0x7f80ec060180 2024-01-21T19:54:18.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.549+0000 7f80f0896700 1 -- 172.21.15.173:0/3917561722 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80ec0f5020 con 0x7f80ec060180 2024-01-21T19:54:18.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.549+0000 7f80d3fff700 1 -- 172.21.15.173:0/3917561722 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80dc013f00 con 0x7f80ec060180 2024-01-21T19:54:18.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.549+0000 7f80d3fff700 1 -- 172.21.15.173:0/3917561722 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80dc014d90 con 0x7f80ec060180 2024-01-21T19:54:18.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.549+0000 7f80f0896700 1 -- 172.21.15.173:0/3917561722 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80ec0f54b0 con 0x7f80ec060180 2024-01-21T19:54:18.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.550+0000 7f80d3fff700 1 -- 172.21.15.173:0/3917561722 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80dc014420 con 0x7f80ec060180 2024-01-21T19:54:18.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.550+0000 7f80d3fff700 1 --2- 172.21.15.173:0/3917561722 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f80d402ff60 0x7f80d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:18.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.550+0000 7f80d3fff700 1 -- 172.21.15.173:0/3917561722 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f80dc0410c0 con 0x7f80ec060180 2024-01-21T19:54:18.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.550+0000 7f80f0896700 1 -- 172.21.15.173:0/3917561722 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80ec059100 con 0x7f80ec060180 2024-01-21T19:54:18.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.550+0000 7f80eaffd700 1 --2- 172.21.15.173:0/3917561722 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f80d402ff60 0x7f80d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:18.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.551+0000 7f80eaffd700 1 --2- 172.21.15.173:0/3917561722 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f80d402ff60 0x7f80d4032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f80e00096a0 tx=0x7f80e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:18.547 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.555+0000 7f80d3fff700 1 -- 172.21.15.173:0/3917561722 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f80dc0095f0 con 0x7f80ec060180 2024-01-21T19:54:18.752 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.760+0000 7f80f0896700 1 -- 172.21.15.173:0/3917561722 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80ec0f5bb0 con 0x7f80ec060180 2024-01-21T19:54:18.753 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.760+0000 7f80d3fff700 1 -- 172.21.15.173:0/3917561722 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f80dc01db70 con 0x7f80ec060180 2024-01-21T19:54:18.753 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:54:18.753 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:54:18.754 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.762+0000 7f80f0896700 1 -- 172.21.15.173:0/3917561722 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f80d402ff60 msgr2=0x7f80d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:18.754 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.762+0000 7f80f0896700 1 --2- 172.21.15.173:0/3917561722 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f80d402ff60 0x7f80d4032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f80e00096a0 tx=0x7f80e0006b40).stop 2024-01-21T19:54:18.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.762+0000 7f80f0896700 1 -- 172.21.15.173:0/3917561722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80ec060180 msgr2=0x7f80ec0f6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:18.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.762+0000 7f80f0896700 1 --2- 172.21.15.173:0/3917561722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80ec060180 0x7f80ec0f6a00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f80dc013510 tx=0x7f80dc00b040).stop 2024-01-21T19:54:18.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.762+0000 7f80f0896700 1 -- 172.21.15.173:0/3917561722 shutdown_connections 2024-01-21T19:54:18.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.762+0000 7f80f0896700 1 --2- 172.21.15.173:0/3917561722 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f80d402ff60 0x7f80d4032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:18.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.762+0000 7f80f0896700 1 --2- 172.21.15.173:0/3917561722 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80ec060180 0x7f80ec0f6a00 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:18.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.762+0000 7f80f0896700 1 -- 172.21.15.173:0/3917561722 >> 172.21.15.173:0/3917561722 conn(0x7f80ec0f09a0 msgr2=0x7f80ec0f1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:18.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.762+0000 7f80f0896700 1 -- 172.21.15.173:0/3917561722 shutdown_connections 2024-01-21T19:54:18.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:18.762+0000 7f80f0896700 1 -- 172.21.15.173:0/3917561722 wait complete. 2024-01-21T19:54:18.755 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:54:19.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:18 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:16.987168+0000 mgr.smithi169.espgue (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:20.136 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:20.137 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:54:20.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:19 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:18.762141+0000 mon.smithi169 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.173:0/3917561722' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:21.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:20 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:18.987415+0000 mgr.smithi169.espgue (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:21.303 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.310+0000 7f3234de6700 1 -- 172.21.15.173:0/624421384 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32300f4880 msgr2=0x7f32300f4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:21.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.310+0000 7f3234de6700 1 --2- 172.21.15.173:0/624421384 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32300f4880 0x7f32300f4ca0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f3224009fb0 tx=0x7f3224009b30).stop 2024-01-21T19:54:21.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.310+0000 7f3234de6700 1 -- 172.21.15.173:0/624421384 shutdown_connections 2024-01-21T19:54:21.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.310+0000 7f3234de6700 1 --2- 172.21.15.173:0/624421384 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32300f4880 0x7f32300f4ca0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:21.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.310+0000 7f3234de6700 1 -- 172.21.15.173:0/624421384 >> 172.21.15.173:0/624421384 conn(0x7f32300f02e0 msgr2=0x7f32300f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:21.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.310+0000 7f3234de6700 1 -- 172.21.15.173:0/624421384 shutdown_connections 2024-01-21T19:54:21.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.310+0000 7f3234de6700 1 -- 172.21.15.173:0/624421384 wait complete. 2024-01-21T19:54:21.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.311+0000 7f3234de6700 1 Processor -- start 2024-01-21T19:54:21.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.311+0000 7f3234de6700 1 -- start start 2024-01-21T19:54:21.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.311+0000 7f3234de6700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32300f4880 0x7f32301066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:21.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.311+0000 7f3234de6700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3230106ba0 con 0x7f32300f4880 2024-01-21T19:54:21.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.311+0000 7f322e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32300f4880 0x7f32301066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:21.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.311+0000 7f322e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32300f4880 0x7f32301066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:55060/0 (socket says 172.21.15.173:55060) 2024-01-21T19:54:21.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.311+0000 7f322e59c700 1 -- 172.21.15.173:0/2619756924 learned_addr learned my addr 172.21.15.173:0/2619756924 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:54:21.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.312+0000 7f322e59c700 1 -- 172.21.15.173:0/2619756924 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3224005040 con 0x7f32300f4880 2024-01-21T19:54:21.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.312+0000 7f322e59c700 1 --2- 172.21.15.173:0/2619756924 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32300f4880 0x7f32301066b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f3224013510 tx=0x7f322400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:21.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.312+0000 7f321effd700 1 -- 172.21.15.173:0/2619756924 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3224009320 con 0x7f32300f4880 2024-01-21T19:54:21.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.312+0000 7f3234de6700 1 -- 172.21.15.173:0/2619756924 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3230106da0 con 0x7f32300f4880 2024-01-21T19:54:21.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.312+0000 7f321effd700 1 -- 172.21.15.173:0/2619756924 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3224013f00 con 0x7f32300f4880 2024-01-21T19:54:21.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.312+0000 7f321effd700 1 -- 172.21.15.173:0/2619756924 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3224014dc0 con 0x7f32300f4880 2024-01-21T19:54:21.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.312+0000 7f3234de6700 1 -- 172.21.15.173:0/2619756924 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3230107230 con 0x7f32300f4880 2024-01-21T19:54:21.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.313+0000 7f321effd700 1 -- 172.21.15.173:0/2619756924 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3224014420 con 0x7f32300f4880 2024-01-21T19:54:21.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.313+0000 7f3234de6700 1 -- 172.21.15.173:0/2619756924 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3230045ad0 con 0x7f32300f4880 2024-01-21T19:54:21.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.313+0000 7f321effd700 1 --2- 172.21.15.173:0/2619756924 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f321802ff60 0x7f3218032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:21.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.313+0000 7f321effd700 1 -- 172.21.15.173:0/2619756924 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3224041050 con 0x7f32300f4880 2024-01-21T19:54:21.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.313+0000 7f322dd9b700 1 --2- 172.21.15.173:0/2619756924 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f321802ff60 0x7f3218032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:21.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.314+0000 7f322dd9b700 1 --2- 172.21.15.173:0/2619756924 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f321802ff60 0x7f3218032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f32200096a0 tx=0x7f3220006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:21.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.317+0000 7f321effd700 1 -- 172.21.15.173:0/2619756924 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32240095f0 con 0x7f32300f4880 2024-01-21T19:54:21.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.522+0000 7f3234de6700 1 -- 172.21.15.173:0/2619756924 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f32300fbf50 con 0x7f32300f4880 2024-01-21T19:54:21.515 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.522+0000 7f321effd700 1 -- 172.21.15.173:0/2619756924 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f322401cb10 con 0x7f32300f4880 2024-01-21T19:54:21.515 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:54:21.515 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:54:21.516 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.524+0000 7f3234de6700 1 -- 172.21.15.173:0/2619756924 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f321802ff60 msgr2=0x7f3218032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:21.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.524+0000 7f3234de6700 1 --2- 172.21.15.173:0/2619756924 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f321802ff60 0x7f3218032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f32200096a0 tx=0x7f3220006b40).stop 2024-01-21T19:54:21.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.524+0000 7f3234de6700 1 -- 172.21.15.173:0/2619756924 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32300f4880 msgr2=0x7f32301066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:21.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.524+0000 7f3234de6700 1 --2- 172.21.15.173:0/2619756924 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32300f4880 0x7f32301066b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f3224013510 tx=0x7f322400b040).stop 2024-01-21T19:54:21.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.524+0000 7f3234de6700 1 -- 172.21.15.173:0/2619756924 shutdown_connections 2024-01-21T19:54:21.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.524+0000 7f3234de6700 1 --2- 172.21.15.173:0/2619756924 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f321802ff60 0x7f3218032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:21.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.524+0000 7f3234de6700 1 --2- 172.21.15.173:0/2619756924 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32300f4880 0x7f32301066b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:21.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.524+0000 7f3234de6700 1 -- 172.21.15.173:0/2619756924 >> 172.21.15.173:0/2619756924 conn(0x7f32300f02e0 msgr2=0x7f32300f0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:21.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.524+0000 7f3234de6700 1 -- 172.21.15.173:0/2619756924 shutdown_connections 2024-01-21T19:54:21.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:21.524+0000 7f3234de6700 1 -- 172.21.15.173:0/2619756924 wait complete. 2024-01-21T19:54:21.517 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:54:22.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:21 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:21.524184+0000 mon.smithi169 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.173:0/2619756924' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:22.907 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:22.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:54:23.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:22 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:20.987749+0000 mgr.smithi169.espgue (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:24.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.097+0000 7f68655a2700 1 -- 172.21.15.173:0/3196112944 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68600f5410 msgr2=0x7f68600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:24.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.097+0000 7f68655a2700 1 --2- 172.21.15.173:0/3196112944 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68600f5410 0x7f68600f5830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f6848009fb0 tx=0x7f6848009b30).stop 2024-01-21T19:54:24.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.097+0000 7f68655a2700 1 -- 172.21.15.173:0/3196112944 shutdown_connections 2024-01-21T19:54:24.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.097+0000 7f68655a2700 1 --2- 172.21.15.173:0/3196112944 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68600f5410 0x7f68600f5830 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:24.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.097+0000 7f68655a2700 1 -- 172.21.15.173:0/3196112944 >> 172.21.15.173:0/3196112944 conn(0x7f68600f0b70 msgr2=0x7f68600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:24.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.097+0000 7f68655a2700 1 -- 172.21.15.173:0/3196112944 shutdown_connections 2024-01-21T19:54:24.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.097+0000 7f68655a2700 1 -- 172.21.15.173:0/3196112944 wait complete. 2024-01-21T19:54:24.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.098+0000 7f68655a2700 1 Processor -- start 2024-01-21T19:54:24.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.098+0000 7f68655a2700 1 -- start start 2024-01-21T19:54:24.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.098+0000 7f68655a2700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68600f5410 0x7f68601093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:24.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.098+0000 7f68655a2700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6860060280 con 0x7f68600f5410 2024-01-21T19:54:24.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.098+0000 7f685effd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68600f5410 0x7f68601093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:24.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.098+0000 7f685effd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68600f5410 0x7f68601093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:56240/0 (socket says 172.21.15.173:56240) 2024-01-21T19:54:24.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.098+0000 7f685effd700 1 -- 172.21.15.173:0/1049596286 learned_addr learned my addr 172.21.15.173:0/1049596286 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:54:24.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.099+0000 7f685effd700 1 -- 172.21.15.173:0/1049596286 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6848005040 con 0x7f68600f5410 2024-01-21T19:54:24.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.099+0000 7f685effd700 1 --2- 172.21.15.173:0/1049596286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68600f5410 0x7f68601093c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f6848013510 tx=0x7f684800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:24.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.099+0000 7f68577fe700 1 -- 172.21.15.173:0/1049596286 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6848009320 con 0x7f68600f5410 2024-01-21T19:54:24.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.099+0000 7f68655a2700 1 -- 172.21.15.173:0/1049596286 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6860060480 con 0x7f68600f5410 2024-01-21T19:54:24.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.099+0000 7f68577fe700 1 -- 172.21.15.173:0/1049596286 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6848013f00 con 0x7f68600f5410 2024-01-21T19:54:24.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.099+0000 7f68577fe700 1 -- 172.21.15.173:0/1049596286 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6848014d90 con 0x7f68600f5410 2024-01-21T19:54:24.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.099+0000 7f68655a2700 1 -- 172.21.15.173:0/1049596286 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6860109af0 con 0x7f68600f5410 2024-01-21T19:54:24.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.100+0000 7f68577fe700 1 -- 172.21.15.173:0/1049596286 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68480095f0 con 0x7f68600f5410 2024-01-21T19:54:24.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.100+0000 7f68655a2700 1 -- 172.21.15.173:0/1049596286 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6860045ad0 con 0x7f68600f5410 2024-01-21T19:54:24.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.100+0000 7f68577fe700 1 --2- 172.21.15.173:0/1049596286 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f684c02ff60 0x7f684c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:24.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.100+0000 7f68577fe700 1 -- 172.21.15.173:0/1049596286 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6848041000 con 0x7f68600f5410 2024-01-21T19:54:24.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.100+0000 7f685e7fc700 1 --2- 172.21.15.173:0/1049596286 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f684c02ff60 0x7f684c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:24.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.101+0000 7f685e7fc700 1 --2- 172.21.15.173:0/1049596286 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f684c02ff60 0x7f684c032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f68500096a0 tx=0x7f6850006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:24.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.105+0000 7f68577fe700 1 -- 172.21.15.173:0/1049596286 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6848014420 con 0x7f68600f5410 2024-01-21T19:54:24.303 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.311+0000 7f68655a2700 1 -- 172.21.15.173:0/1049596286 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f686010a310 con 0x7f68600f5410 2024-01-21T19:54:24.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.311+0000 7f68577fe700 1 -- 172.21.15.173:0/1049596286 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6848010230 con 0x7f68600f5410 2024-01-21T19:54:24.304 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:54:24.304 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:54:24.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.313+0000 7f68655a2700 1 -- 172.21.15.173:0/1049596286 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f684c02ff60 msgr2=0x7f684c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:24.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.313+0000 7f68655a2700 1 --2- 172.21.15.173:0/1049596286 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f684c02ff60 0x7f684c032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f68500096a0 tx=0x7f6850006b40).stop 2024-01-21T19:54:24.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.313+0000 7f68655a2700 1 -- 172.21.15.173:0/1049596286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68600f5410 msgr2=0x7f68601093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:24.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.313+0000 7f68655a2700 1 --2- 172.21.15.173:0/1049596286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68600f5410 0x7f68601093c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f6848013510 tx=0x7f684800b040).stop 2024-01-21T19:54:24.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.313+0000 7f68655a2700 1 -- 172.21.15.173:0/1049596286 shutdown_connections 2024-01-21T19:54:24.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.313+0000 7f68655a2700 1 --2- 172.21.15.173:0/1049596286 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f684c02ff60 0x7f684c032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:24.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.313+0000 7f68655a2700 1 --2- 172.21.15.173:0/1049596286 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68600f5410 0x7f68601093c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:24.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.313+0000 7f68655a2700 1 -- 172.21.15.173:0/1049596286 >> 172.21.15.173:0/1049596286 conn(0x7f68600f0b70 msgr2=0x7f68600f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:24.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.313+0000 7f68655a2700 1 -- 172.21.15.173:0/1049596286 shutdown_connections 2024-01-21T19:54:24.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:24.313+0000 7f68655a2700 1 -- 172.21.15.173:0/1049596286 wait complete. 2024-01-21T19:54:24.307 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:54:25.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:24 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:22.988082+0000 mgr.smithi169.espgue (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:25.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:24 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:24.313319+0000 mon.smithi169 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.173:0/1049596286' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:25.677 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:25.677 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:54:26.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.879+0000 7f5a34a9e700 1 -- 172.21.15.173:0/449749615 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a30060280 msgr2=0x7f5a300f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:26.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.879+0000 7f5a34a9e700 1 --2- 172.21.15.173:0/449749615 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a30060280 0x7f5a300f90f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f5a20009fb0 tx=0x7f5a20009b30).stop 2024-01-21T19:54:26.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.879+0000 7f5a34a9e700 1 -- 172.21.15.173:0/449749615 shutdown_connections 2024-01-21T19:54:26.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.879+0000 7f5a34a9e700 1 --2- 172.21.15.173:0/449749615 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a30060280 0x7f5a300f90f0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:26.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.879+0000 7f5a34a9e700 1 -- 172.21.15.173:0/449749615 >> 172.21.15.173:0/449749615 conn(0x7f5a300f0b70 msgr2=0x7f5a300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:26.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.879+0000 7f5a34a9e700 1 -- 172.21.15.173:0/449749615 shutdown_connections 2024-01-21T19:54:26.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.879+0000 7f5a34a9e700 1 -- 172.21.15.173:0/449749615 wait complete. 2024-01-21T19:54:26.875 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.880+0000 7f5a34a9e700 1 Processor -- start 2024-01-21T19:54:26.875 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.880+0000 7f5a34a9e700 1 -- start start 2024-01-21T19:54:26.875 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.880+0000 7f5a34a9e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a30060280 0x7f5a301066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:26.875 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.880+0000 7f5a34a9e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a30106bd0 con 0x7f5a30060280 2024-01-21T19:54:26.875 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.881+0000 7f5a2e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a30060280 0x7f5a301066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:26.875 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.881+0000 7f5a2e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a30060280 0x7f5a301066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:56266/0 (socket says 172.21.15.173:56266) 2024-01-21T19:54:26.875 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.881+0000 7f5a2e59c700 1 -- 172.21.15.173:0/1252121115 learned_addr learned my addr 172.21.15.173:0/1252121115 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:54:26.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.881+0000 7f5a2e59c700 1 -- 172.21.15.173:0/1252121115 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a20005040 con 0x7f5a30060280 2024-01-21T19:54:26.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.881+0000 7f5a2e59c700 1 --2- 172.21.15.173:0/1252121115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a30060280 0x7f5a301066e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f5a20013510 tx=0x7f5a2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:26.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.881+0000 7f5a26ffd700 1 -- 172.21.15.173:0/1252121115 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a20009320 con 0x7f5a30060280 2024-01-21T19:54:26.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.881+0000 7f5a34a9e700 1 -- 172.21.15.173:0/1252121115 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a30106dd0 con 0x7f5a30060280 2024-01-21T19:54:26.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.881+0000 7f5a26ffd700 1 -- 172.21.15.173:0/1252121115 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a20013f00 con 0x7f5a30060280 2024-01-21T19:54:26.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.881+0000 7f5a26ffd700 1 -- 172.21.15.173:0/1252121115 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a20014dc0 con 0x7f5a30060280 2024-01-21T19:54:26.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.881+0000 7f5a34a9e700 1 -- 172.21.15.173:0/1252121115 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a30107260 con 0x7f5a30060280 2024-01-21T19:54:26.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.882+0000 7f5a26ffd700 1 -- 172.21.15.173:0/1252121115 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5a20014420 con 0x7f5a30060280 2024-01-21T19:54:26.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.882+0000 7f5a34a9e700 1 -- 172.21.15.173:0/1252121115 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a30045ad0 con 0x7f5a30060280 2024-01-21T19:54:26.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.882+0000 7f5a26ffd700 1 --2- 172.21.15.173:0/1252121115 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5a1c02ffb0 0x7f5a1c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:26.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.882+0000 7f5a26ffd700 1 -- 172.21.15.173:0/1252121115 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5a200410a0 con 0x7f5a30060280 2024-01-21T19:54:26.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.882+0000 7f5a2dd9b700 1 --2- 172.21.15.173:0/1252121115 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5a1c02ffb0 0x7f5a1c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:26.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.883+0000 7f5a2dd9b700 1 --2- 172.21.15.173:0/1252121115 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5a1c02ffb0 0x7f5a1c032470 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f5a180096a0 tx=0x7f5a18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:26.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:26.887+0000 7f5a26ffd700 1 -- 172.21.15.173:0/1252121115 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a2006d2b0 con 0x7f5a30060280 2024-01-21T19:54:27.086 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:27.093+0000 7f5a34a9e700 1 -- 172.21.15.173:0/1252121115 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a301080a0 con 0x7f5a30060280 2024-01-21T19:54:27.087 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:27.094+0000 7f5a26ffd700 1 -- 172.21.15.173:0/1252121115 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5a200095f0 con 0x7f5a30060280 2024-01-21T19:54:27.087 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:54:27.087 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:54:27.088 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:27.096+0000 7f5a34a9e700 1 -- 172.21.15.173:0/1252121115 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5a1c02ffb0 msgr2=0x7f5a1c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:27.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:27.096+0000 7f5a34a9e700 1 --2- 172.21.15.173:0/1252121115 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5a1c02ffb0 0x7f5a1c032470 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f5a180096a0 tx=0x7f5a18006b40).stop 2024-01-21T19:54:27.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:27.096+0000 7f5a34a9e700 1 -- 172.21.15.173:0/1252121115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a30060280 msgr2=0x7f5a301066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:27.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:27.096+0000 7f5a34a9e700 1 --2- 172.21.15.173:0/1252121115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a30060280 0x7f5a301066e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f5a20013510 tx=0x7f5a2000b040).stop 2024-01-21T19:54:27.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:27.096+0000 7f5a34a9e700 1 -- 172.21.15.173:0/1252121115 shutdown_connections 2024-01-21T19:54:27.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:27.096+0000 7f5a34a9e700 1 --2- 172.21.15.173:0/1252121115 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5a1c02ffb0 0x7f5a1c032470 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:27.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:27.096+0000 7f5a34a9e700 1 --2- 172.21.15.173:0/1252121115 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a30060280 0x7f5a301066e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:27.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:27.096+0000 7f5a34a9e700 1 -- 172.21.15.173:0/1252121115 >> 172.21.15.173:0/1252121115 conn(0x7f5a300f0b70 msgr2=0x7f5a300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:27.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:27.096+0000 7f5a34a9e700 1 -- 172.21.15.173:0/1252121115 shutdown_connections 2024-01-21T19:54:27.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:27.096+0000 7f5a34a9e700 1 -- 172.21.15.173:0/1252121115 wait complete. 2024-01-21T19:54:27.089 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:54:27.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:26 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:24.988330+0000 mgr.smithi169.espgue (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:28.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:27 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:27.095785+0000 mon.smithi169 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.173:0/1252121115' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:28.480 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:28.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:54:29.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:28 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:26.988590+0000 mgr.smithi169.espgue (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:29.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.654+0000 7f5f60211700 1 -- 172.21.15.173:0/806759362 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5f58060280 msgr2=0x7f5f580f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:29.648 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.654+0000 7f5f60211700 1 --2- 172.21.15.173:0/806759362 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5f58060280 0x7f5f580f90f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f5f54009fb0 tx=0x7f5f54009b30).stop 2024-01-21T19:54:29.648 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.654+0000 7f5f60211700 1 -- 172.21.15.173:0/806759362 shutdown_connections 2024-01-21T19:54:29.648 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.654+0000 7f5f60211700 1 --2- 172.21.15.173:0/806759362 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5f58060280 0x7f5f580f90f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:29.648 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.654+0000 7f5f60211700 1 -- 172.21.15.173:0/806759362 >> 172.21.15.173:0/806759362 conn(0x7f5f580f0b70 msgr2=0x7f5f580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:29.648 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.654+0000 7f5f60211700 1 -- 172.21.15.173:0/806759362 shutdown_connections 2024-01-21T19:54:29.648 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.654+0000 7f5f60211700 1 -- 172.21.15.173:0/806759362 wait complete. 2024-01-21T19:54:29.648 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.654+0000 7f5f60211700 1 Processor -- start 2024-01-21T19:54:29.648 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.655+0000 7f5f60211700 1 -- start start 2024-01-21T19:54:29.648 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.655+0000 7f5f60211700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5f58060280 0x7f5f580f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:29.648 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.655+0000 7f5f60211700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f580f4ff0 con 0x7f5f58060280 2024-01-21T19:54:29.648 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.655+0000 7f5f5dfad700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5f58060280 0x7f5f580f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:29.648 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.655+0000 7f5f5dfad700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5f58060280 0x7f5f580f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:56280/0 (socket says 172.21.15.173:56280) 2024-01-21T19:54:29.649 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.655+0000 7f5f5dfad700 1 -- 172.21.15.173:0/791631603 learned_addr learned my addr 172.21.15.173:0/791631603 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:54:29.649 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.655+0000 7f5f5dfad700 1 -- 172.21.15.173:0/791631603 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f54005040 con 0x7f5f58060280 2024-01-21T19:54:29.649 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.655+0000 7f5f5dfad700 1 --2- 172.21.15.173:0/791631603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5f58060280 0x7f5f580f6b70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f5f54013510 tx=0x7f5f5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:29.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.655+0000 7f5f4e7fc700 1 -- 172.21.15.173:0/791631603 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f54009320 con 0x7f5f58060280 2024-01-21T19:54:29.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.656+0000 7f5f60211700 1 -- 172.21.15.173:0/791631603 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f580f5190 con 0x7f5f58060280 2024-01-21T19:54:29.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.656+0000 7f5f4e7fc700 1 -- 172.21.15.173:0/791631603 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f54013f00 con 0x7f5f58060280 2024-01-21T19:54:29.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.656+0000 7f5f4e7fc700 1 -- 172.21.15.173:0/791631603 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f54014dc0 con 0x7f5f58060280 2024-01-21T19:54:29.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.656+0000 7f5f60211700 1 -- 172.21.15.173:0/791631603 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f580f5620 con 0x7f5f58060280 2024-01-21T19:54:29.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.656+0000 7f5f4e7fc700 1 -- 172.21.15.173:0/791631603 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5f540095f0 con 0x7f5f58060280 2024-01-21T19:54:29.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.656+0000 7f5f60211700 1 -- 172.21.15.173:0/791631603 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f58059100 con 0x7f5f58060280 2024-01-21T19:54:29.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.656+0000 7f5f4e7fc700 1 --2- 172.21.15.173:0/791631603 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5f4402ff60 0x7f5f44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:29.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.657+0000 7f5f4e7fc700 1 -- 172.21.15.173:0/791631603 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5f540410c0 con 0x7f5f58060280 2024-01-21T19:54:29.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.657+0000 7f5f5d7ac700 1 --2- 172.21.15.173:0/791631603 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5f4402ff60 0x7f5f44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:29.651 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.657+0000 7f5f5d7ac700 1 --2- 172.21.15.173:0/791631603 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5f4402ff60 0x7f5f44032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f5f480096a0 tx=0x7f5f48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:29.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.661+0000 7f5f4e7fc700 1 -- 172.21.15.173:0/791631603 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f5401c790 con 0x7f5f58060280 2024-01-21T19:54:29.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.867+0000 7f5f60211700 1 -- 172.21.15.173:0/791631603 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f580fbf50 con 0x7f5f58060280 2024-01-21T19:54:29.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.868+0000 7f5f4e7fc700 1 -- 172.21.15.173:0/791631603 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5f54010230 con 0x7f5f58060280 2024-01-21T19:54:29.861 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:54:29.861 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:54:29.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.870+0000 7f5f60211700 1 -- 172.21.15.173:0/791631603 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5f4402ff60 msgr2=0x7f5f44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:29.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.870+0000 7f5f60211700 1 --2- 172.21.15.173:0/791631603 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5f4402ff60 0x7f5f44032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f5f480096a0 tx=0x7f5f48006b40).stop 2024-01-21T19:54:29.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.870+0000 7f5f60211700 1 -- 172.21.15.173:0/791631603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5f58060280 msgr2=0x7f5f580f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:29.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.870+0000 7f5f60211700 1 --2- 172.21.15.173:0/791631603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5f58060280 0x7f5f580f6b70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f5f54013510 tx=0x7f5f5400b040).stop 2024-01-21T19:54:29.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.870+0000 7f5f60211700 1 -- 172.21.15.173:0/791631603 shutdown_connections 2024-01-21T19:54:29.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.870+0000 7f5f60211700 1 --2- 172.21.15.173:0/791631603 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5f4402ff60 0x7f5f44032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:29.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.870+0000 7f5f60211700 1 --2- 172.21.15.173:0/791631603 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5f58060280 0x7f5f580f6b70 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:29.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.870+0000 7f5f60211700 1 -- 172.21.15.173:0/791631603 >> 172.21.15.173:0/791631603 conn(0x7f5f580f0b70 msgr2=0x7f5f580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:29.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.870+0000 7f5f60211700 1 -- 172.21.15.173:0/791631603 shutdown_connections 2024-01-21T19:54:29.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:29.870+0000 7f5f60211700 1 -- 172.21.15.173:0/791631603 wait complete. 2024-01-21T19:54:29.863 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:54:31.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:30 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:28.988931+0000 mgr.smithi169.espgue (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:31.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:30 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:29.869765+0000 mon.smithi169 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.173:0/791631603' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:31.299 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:31.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:54:32.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:31.074917+0000 mon.smithi169 (mon.0) 253 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/mirror_snapshot_schedule"}]: dispatch 2024-01-21T19:54:32.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:31.078378+0000 mon.smithi169 (mon.0) 254 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/trash_purge_schedule"}]: dispatch 2024-01-21T19:54:32.453 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.459+0000 7fd9e89f5700 1 -- 172.21.15.173:0/3481270563 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9e4060200 msgr2=0x7fd9e40fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:32.453 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.459+0000 7fd9e89f5700 1 --2- 172.21.15.173:0/3481270563 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9e4060200 0x7fd9e40fb570 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fd9cc009fb0 tx=0x7fd9cc009b30).stop 2024-01-21T19:54:32.454 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.459+0000 7fd9e89f5700 1 -- 172.21.15.173:0/3481270563 shutdown_connections 2024-01-21T19:54:32.454 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.459+0000 7fd9e89f5700 1 --2- 172.21.15.173:0/3481270563 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9e4060200 0x7fd9e40fb570 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:32.454 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.459+0000 7fd9e89f5700 1 -- 172.21.15.173:0/3481270563 >> 172.21.15.173:0/3481270563 conn(0x7fd9e40f0b70 msgr2=0x7fd9e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:32.454 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.459+0000 7fd9e89f5700 1 -- 172.21.15.173:0/3481270563 shutdown_connections 2024-01-21T19:54:32.454 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.459+0000 7fd9e89f5700 1 -- 172.21.15.173:0/3481270563 wait complete. 2024-01-21T19:54:32.454 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.460+0000 7fd9e89f5700 1 Processor -- start 2024-01-21T19:54:32.454 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.460+0000 7fd9e89f5700 1 -- start start 2024-01-21T19:54:32.454 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.460+0000 7fd9e89f5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9e4060200 0x7fd9e41088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:32.454 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.460+0000 7fd9e89f5700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e4108dc0 con 0x7fd9e4060200 2024-01-21T19:54:32.454 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.460+0000 7fd9e259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9e4060200 0x7fd9e41088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:32.454 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.460+0000 7fd9e259c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9e4060200 0x7fd9e41088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:56306/0 (socket says 172.21.15.173:56306) 2024-01-21T19:54:32.454 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.460+0000 7fd9e259c700 1 -- 172.21.15.173:0/2167973239 learned_addr learned my addr 172.21.15.173:0/2167973239 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:54:32.454 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.460+0000 7fd9e259c700 1 -- 172.21.15.173:0/2167973239 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9cc005040 con 0x7fd9e4060200 2024-01-21T19:54:32.454 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.461+0000 7fd9e259c700 1 --2- 172.21.15.173:0/2167973239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9e4060200 0x7fd9e41088d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fd9cc013510 tx=0x7fd9cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:32.455 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.461+0000 7fd9daffd700 1 -- 172.21.15.173:0/2167973239 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9cc009320 con 0x7fd9e4060200 2024-01-21T19:54:32.455 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.461+0000 7fd9e89f5700 1 -- 172.21.15.173:0/2167973239 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e4108fc0 con 0x7fd9e4060200 2024-01-21T19:54:32.455 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.461+0000 7fd9daffd700 1 -- 172.21.15.173:0/2167973239 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd9cc013f00 con 0x7fd9e4060200 2024-01-21T19:54:32.455 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.461+0000 7fd9daffd700 1 -- 172.21.15.173:0/2167973239 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9cc014dc0 con 0x7fd9e4060200 2024-01-21T19:54:32.455 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.461+0000 7fd9e89f5700 1 -- 172.21.15.173:0/2167973239 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e4109450 con 0x7fd9e4060200 2024-01-21T19:54:32.456 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.462+0000 7fd9daffd700 1 -- 172.21.15.173:0/2167973239 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9cc01d3d0 con 0x7fd9e4060200 2024-01-21T19:54:32.456 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.462+0000 7fd9daffd700 1 --2- 172.21.15.173:0/2167973239 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd9d002ff60 0x7fd9d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:32.456 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.462+0000 7fd9daffd700 1 -- 172.21.15.173:0/2167973239 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd9cc041340 con 0x7fd9e4060200 2024-01-21T19:54:32.456 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.462+0000 7fd9e89f5700 1 -- 172.21.15.173:0/2167973239 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e4045ad0 con 0x7fd9e4060200 2024-01-21T19:54:32.456 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.462+0000 7fd9e1d9b700 1 --2- 172.21.15.173:0/2167973239 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd9d002ff60 0x7fd9d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:32.456 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.462+0000 7fd9e1d9b700 1 --2- 172.21.15.173:0/2167973239 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd9d002ff60 0x7fd9d0032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fd9d40096a0 tx=0x7fd9d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:32.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.466+0000 7fd9daffd700 1 -- 172.21.15.173:0/2167973239 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd9cc012920 con 0x7fd9e4060200 2024-01-21T19:54:32.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.673+0000 7fd9e89f5700 1 -- 172.21.15.173:0/2167973239 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9e410a300 con 0x7fd9e4060200 2024-01-21T19:54:32.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.675+0000 7fd9daffd700 1 -- 172.21.15.173:0/2167973239 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd9cc02b030 con 0x7fd9e4060200 2024-01-21T19:54:32.666 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:54:32.667 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:54:32.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.675+0000 7fd9e89f5700 1 -- 172.21.15.173:0/2167973239 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd9d002ff60 msgr2=0x7fd9d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:32.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.675+0000 7fd9e89f5700 1 --2- 172.21.15.173:0/2167973239 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd9d002ff60 0x7fd9d0032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fd9d40096a0 tx=0x7fd9d4006b40).stop 2024-01-21T19:54:32.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.675+0000 7fd9e89f5700 1 -- 172.21.15.173:0/2167973239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9e4060200 msgr2=0x7fd9e41088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:32.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.675+0000 7fd9e89f5700 1 --2- 172.21.15.173:0/2167973239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9e4060200 0x7fd9e41088d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fd9cc013510 tx=0x7fd9cc00b040).stop 2024-01-21T19:54:32.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.675+0000 7fd9e89f5700 1 -- 172.21.15.173:0/2167973239 shutdown_connections 2024-01-21T19:54:32.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.675+0000 7fd9e89f5700 1 --2- 172.21.15.173:0/2167973239 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fd9d002ff60 0x7fd9d0032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:32.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.675+0000 7fd9e89f5700 1 --2- 172.21.15.173:0/2167973239 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd9e4060200 0x7fd9e41088d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:32.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.675+0000 7fd9e89f5700 1 -- 172.21.15.173:0/2167973239 >> 172.21.15.173:0/2167973239 conn(0x7fd9e40f0b70 msgr2=0x7fd9e40f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:32.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.675+0000 7fd9e89f5700 1 -- 172.21.15.173:0/2167973239 shutdown_connections 2024-01-21T19:54:32.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:32.675+0000 7fd9e89f5700 1 -- 172.21.15.173:0/2167973239 wait complete. 2024-01-21T19:54:32.669 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:54:32.914 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:32 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:30.989274+0000 mgr.smithi169.espgue (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:32.915 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:32 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:32.675587+0000 mon.smithi169 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.173:0/2167973239' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:34.059 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:34.060 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:54:35.167 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:34 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:32.989625+0000 mgr.smithi169.espgue (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:35.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.216+0000 7fe922d23700 1 -- 172.21.15.173:0/1085938073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe91c0f5410 msgr2=0x7fe91c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:35.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.216+0000 7fe922d23700 1 --2- 172.21.15.173:0/1085938073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe91c0f5410 0x7fe91c0f5830 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fe910009fb0 tx=0x7fe910009b30).stop 2024-01-21T19:54:35.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.216+0000 7fe922d23700 1 -- 172.21.15.173:0/1085938073 shutdown_connections 2024-01-21T19:54:35.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.216+0000 7fe922d23700 1 --2- 172.21.15.173:0/1085938073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe91c0f5410 0x7fe91c0f5830 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:35.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.216+0000 7fe922d23700 1 -- 172.21.15.173:0/1085938073 >> 172.21.15.173:0/1085938073 conn(0x7fe91c0f0b70 msgr2=0x7fe91c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:35.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.216+0000 7fe922d23700 1 -- 172.21.15.173:0/1085938073 shutdown_connections 2024-01-21T19:54:35.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.216+0000 7fe922d23700 1 -- 172.21.15.173:0/1085938073 wait complete. 2024-01-21T19:54:35.210 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.216+0000 7fe922d23700 1 Processor -- start 2024-01-21T19:54:35.210 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.216+0000 7fe922d23700 1 -- start start 2024-01-21T19:54:35.210 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.217+0000 7fe922d23700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe91c0f5410 0x7fe91c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:35.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.217+0000 7fe922d23700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c105b20 con 0x7fe91c0f5410 2024-01-21T19:54:35.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.217+0000 7fe920abf700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe91c0f5410 0x7fe91c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:35.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.217+0000 7fe920abf700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe91c0f5410 0x7fe91c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:43344/0 (socket says 172.21.15.173:43344) 2024-01-21T19:54:35.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.217+0000 7fe920abf700 1 -- 172.21.15.173:0/4293649988 learned_addr learned my addr 172.21.15.173:0/4293649988 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:54:35.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.217+0000 7fe920abf700 1 -- 172.21.15.173:0/4293649988 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe910005040 con 0x7fe91c0f5410 2024-01-21T19:54:35.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.217+0000 7fe920abf700 1 --2- 172.21.15.173:0/4293649988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe91c0f5410 0x7fe91c105630 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fe910014000 tx=0x7fe91000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:35.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.218+0000 7fe9197fa700 1 -- 172.21.15.173:0/4293649988 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9100095f0 con 0x7fe91c0f5410 2024-01-21T19:54:35.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.218+0000 7fe922d23700 1 -- 172.21.15.173:0/4293649988 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe91c105d20 con 0x7fe91c0f5410 2024-01-21T19:54:35.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.218+0000 7fe922d23700 1 -- 172.21.15.173:0/4293649988 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe91c1061b0 con 0x7fe91c0f5410 2024-01-21T19:54:35.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.218+0000 7fe9197fa700 1 -- 172.21.15.173:0/4293649988 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe910007e70 con 0x7fe91c0f5410 2024-01-21T19:54:35.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.218+0000 7fe9197fa700 1 -- 172.21.15.173:0/4293649988 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe910014e50 con 0x7fe91c0f5410 2024-01-21T19:54:35.212 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.219+0000 7fe9197fa700 1 -- 172.21.15.173:0/4293649988 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe910025070 con 0x7fe91c0f5410 2024-01-21T19:54:35.212 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.219+0000 7fe922d23700 1 -- 172.21.15.173:0/4293649988 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8fc004fa0 con 0x7fe91c0f5410 2024-01-21T19:54:35.212 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.219+0000 7fe9197fa700 1 --2- 172.21.15.173:0/4293649988 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe90c02ff10 0x7fe90c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:35.212 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.219+0000 7fe9197fa700 1 -- 172.21.15.173:0/4293649988 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe910040970 con 0x7fe91c0f5410 2024-01-21T19:54:35.212 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.219+0000 7fe91bfff700 1 --2- 172.21.15.173:0/4293649988 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe90c02ff10 0x7fe90c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:35.213 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.219+0000 7fe91bfff700 1 --2- 172.21.15.173:0/4293649988 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe90c02ff10 0x7fe90c0323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fe9080096a0 tx=0x7fe908006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:35.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.223+0000 7fe9197fa700 1 -- 172.21.15.173:0/4293649988 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe9100079c0 con 0x7fe91c0f5410 2024-01-21T19:54:35.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.433+0000 7fe922d23700 1 -- 172.21.15.173:0/4293649988 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8fc005670 con 0x7fe91c0f5410 2024-01-21T19:54:35.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.434+0000 7fe9197fa700 1 -- 172.21.15.173:0/4293649988 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe91000f300 con 0x7fe91c0f5410 2024-01-21T19:54:35.427 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:54:35.427 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:54:35.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.435+0000 7fe922d23700 1 -- 172.21.15.173:0/4293649988 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe90c02ff10 msgr2=0x7fe90c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:35.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.435+0000 7fe922d23700 1 --2- 172.21.15.173:0/4293649988 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe90c02ff10 0x7fe90c0323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fe9080096a0 tx=0x7fe908006b40).stop 2024-01-21T19:54:35.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.435+0000 7fe922d23700 1 -- 172.21.15.173:0/4293649988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe91c0f5410 msgr2=0x7fe91c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:35.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.435+0000 7fe922d23700 1 --2- 172.21.15.173:0/4293649988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe91c0f5410 0x7fe91c105630 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fe910014000 tx=0x7fe91000b040).stop 2024-01-21T19:54:35.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.436+0000 7fe922d23700 1 -- 172.21.15.173:0/4293649988 shutdown_connections 2024-01-21T19:54:35.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.436+0000 7fe922d23700 1 --2- 172.21.15.173:0/4293649988 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe90c02ff10 0x7fe90c0323d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:35.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.436+0000 7fe922d23700 1 --2- 172.21.15.173:0/4293649988 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe91c0f5410 0x7fe91c105630 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:35.429 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.436+0000 7fe922d23700 1 -- 172.21.15.173:0/4293649988 >> 172.21.15.173:0/4293649988 conn(0x7fe91c0f0b70 msgr2=0x7fe91c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:35.429 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.436+0000 7fe922d23700 1 -- 172.21.15.173:0/4293649988 shutdown_connections 2024-01-21T19:54:35.429 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:35.436+0000 7fe922d23700 1 -- 172.21.15.173:0/4293649988 wait complete. 2024-01-21T19:54:35.429 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:54:36.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:35 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:35.435665+0000 mon.smithi169 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.173:0/4293649988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:36.821 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:36.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:54:37.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:36 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:34.990056+0000 mgr.smithi169.espgue (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:37.975 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.983+0000 7f3b61e06700 1 -- 172.21.15.173:0/752378336 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b5c0f6510 msgr2=0x7f3b5c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:37.975 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.983+0000 7f3b61e06700 1 --2- 172.21.15.173:0/752378336 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b5c0f6510 0x7f3b5c0f6930 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f3b50009fb0 tx=0x7f3b50009b30).stop 2024-01-21T19:54:37.975 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.984+0000 7f3b61e06700 1 -- 172.21.15.173:0/752378336 shutdown_connections 2024-01-21T19:54:37.975 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.984+0000 7f3b61e06700 1 --2- 172.21.15.173:0/752378336 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b5c0f6510 0x7f3b5c0f6930 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:37.975 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.984+0000 7f3b61e06700 1 -- 172.21.15.173:0/752378336 >> 172.21.15.173:0/752378336 conn(0x7f3b5c0f1c70 msgr2=0x7f3b5c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:37.975 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.984+0000 7f3b61e06700 1 -- 172.21.15.173:0/752378336 shutdown_connections 2024-01-21T19:54:37.976 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.984+0000 7f3b61e06700 1 -- 172.21.15.173:0/752378336 wait complete. 2024-01-21T19:54:37.977 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.984+0000 7f3b61e06700 1 Processor -- start 2024-01-21T19:54:37.977 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.984+0000 7f3b61e06700 1 -- start start 2024-01-21T19:54:37.977 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.984+0000 7f3b61e06700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b5c106840 0x7f3b5c106c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:37.977 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.984+0000 7f3b61e06700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c107150 con 0x7f3b5c106840 2024-01-21T19:54:37.977 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.985+0000 7f3b5b7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b5c106840 0x7f3b5c106c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:37.977 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.985+0000 7f3b5b7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b5c106840 0x7f3b5c106c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:43372/0 (socket says 172.21.15.173:43372) 2024-01-21T19:54:37.977 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.985+0000 7f3b5b7fe700 1 -- 172.21.15.173:0/971115810 learned_addr learned my addr 172.21.15.173:0/971115810 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:54:37.977 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.985+0000 7f3b5b7fe700 1 -- 172.21.15.173:0/971115810 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b50005040 con 0x7f3b5c106840 2024-01-21T19:54:37.978 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.985+0000 7f3b5b7fe700 1 --2- 172.21.15.173:0/971115810 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b5c106840 0x7f3b5c106c60 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f3b50009f80 tx=0x7f3b50009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:37.978 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.985+0000 7f3b43fff700 1 -- 172.21.15.173:0/971115810 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b500136c0 con 0x7f3b5c106840 2024-01-21T19:54:37.978 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.985+0000 7f3b43fff700 1 -- 172.21.15.173:0/971115810 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3b500095f0 con 0x7f3b5c106840 2024-01-21T19:54:37.978 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.985+0000 7f3b61e06700 1 -- 172.21.15.173:0/971115810 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b5c107350 con 0x7f3b5c106840 2024-01-21T19:54:37.978 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.985+0000 7f3b43fff700 1 -- 172.21.15.173:0/971115810 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b50014420 con 0x7f3b5c106840 2024-01-21T19:54:37.978 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.985+0000 7f3b61e06700 1 -- 172.21.15.173:0/971115810 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b5c107ea0 con 0x7f3b5c106840 2024-01-21T19:54:37.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.986+0000 7f3b43fff700 1 -- 172.21.15.173:0/971115810 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3b50025060 con 0x7f3b5c106840 2024-01-21T19:54:37.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.986+0000 7f3b61e06700 1 -- 172.21.15.173:0/971115810 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b5c059100 con 0x7f3b5c106840 2024-01-21T19:54:37.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.986+0000 7f3b43fff700 1 --2- 172.21.15.173:0/971115810 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3b4402ff60 0x7f3b44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:37.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.986+0000 7f3b43fff700 1 -- 172.21.15.173:0/971115810 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3b50040d80 con 0x7f3b5c106840 2024-01-21T19:54:37.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.987+0000 7f3b5affd700 1 --2- 172.21.15.173:0/971115810 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3b4402ff60 0x7f3b44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:37.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.987+0000 7f3b5affd700 1 --2- 172.21.15.173:0/971115810 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3b4402ff60 0x7f3b44032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f3b4c0096a0 tx=0x7f3b4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:37.982 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:37.991+0000 7f3b43fff700 1 -- 172.21.15.173:0/971115810 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3b5001cb60 con 0x7f3b5c106840 2024-01-21T19:54:38.192 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:38.199+0000 7f3b61e06700 1 -- 172.21.15.173:0/971115810 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b5c108660 con 0x7f3b5c106840 2024-01-21T19:54:38.192 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:38.200+0000 7f3b43fff700 1 -- 172.21.15.173:0/971115810 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3b50042070 con 0x7f3b5c106840 2024-01-21T19:54:38.192 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:54:38.192 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:54:38.193 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:38.202+0000 7f3b61e06700 1 -- 172.21.15.173:0/971115810 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3b4402ff60 msgr2=0x7f3b44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:38.193 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:38.202+0000 7f3b61e06700 1 --2- 172.21.15.173:0/971115810 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3b4402ff60 0x7f3b44032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f3b4c0096a0 tx=0x7f3b4c006b40).stop 2024-01-21T19:54:38.193 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:38.202+0000 7f3b61e06700 1 -- 172.21.15.173:0/971115810 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b5c106840 msgr2=0x7f3b5c106c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:38.193 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:38.202+0000 7f3b61e06700 1 --2- 172.21.15.173:0/971115810 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b5c106840 0x7f3b5c106c60 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f3b50009f80 tx=0x7f3b50009b30).stop 2024-01-21T19:54:38.194 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:38.202+0000 7f3b61e06700 1 -- 172.21.15.173:0/971115810 shutdown_connections 2024-01-21T19:54:38.194 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:38.202+0000 7f3b61e06700 1 --2- 172.21.15.173:0/971115810 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3b4402ff60 0x7f3b44032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:38.194 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:38.202+0000 7f3b61e06700 1 --2- 172.21.15.173:0/971115810 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b5c106840 0x7f3b5c106c60 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:38.194 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:38.202+0000 7f3b61e06700 1 -- 172.21.15.173:0/971115810 >> 172.21.15.173:0/971115810 conn(0x7f3b5c0f1c70 msgr2=0x7f3b5c0f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:38.194 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:38.202+0000 7f3b61e06700 1 -- 172.21.15.173:0/971115810 shutdown_connections 2024-01-21T19:54:38.194 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:38.202+0000 7f3b61e06700 1 -- 172.21.15.173:0/971115810 wait complete. 2024-01-21T19:54:38.194 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:54:39.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:36.990348+0000 mgr.smithi169.espgue (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:39.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:38.200858+0000 mon.smithi169 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.173:0/971115810' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:39.591 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:39.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:54:40.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.784+0000 7fdf9599a700 1 -- 172.21.15.173:0/2651991328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf900f5410 msgr2=0x7fdf900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:40.776 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.784+0000 7fdf9599a700 1 --2- 172.21.15.173:0/2651991328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf900f5410 0x7fdf900f5830 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fdf80009fb0 tx=0x7fdf80009b30).stop 2024-01-21T19:54:40.776 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.784+0000 7fdf9599a700 1 -- 172.21.15.173:0/2651991328 shutdown_connections 2024-01-21T19:54:40.776 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.784+0000 7fdf9599a700 1 --2- 172.21.15.173:0/2651991328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf900f5410 0x7fdf900f5830 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:40.776 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.784+0000 7fdf9599a700 1 -- 172.21.15.173:0/2651991328 >> 172.21.15.173:0/2651991328 conn(0x7fdf900f0b70 msgr2=0x7fdf900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:40.776 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.784+0000 7fdf9599a700 1 -- 172.21.15.173:0/2651991328 shutdown_connections 2024-01-21T19:54:40.776 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.784+0000 7fdf9599a700 1 -- 172.21.15.173:0/2651991328 wait complete. 2024-01-21T19:54:40.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.785+0000 7fdf9599a700 1 Processor -- start 2024-01-21T19:54:40.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.785+0000 7fdf9599a700 1 -- start start 2024-01-21T19:54:40.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.785+0000 7fdf9599a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf900f5410 0x7fdf90106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:40.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.785+0000 7fdf9599a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf90106bf0 con 0x7fdf900f5410 2024-01-21T19:54:40.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.785+0000 7fdf8effd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf900f5410 0x7fdf90106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:40.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.785+0000 7fdf8effd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf900f5410 0x7fdf90106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:43390/0 (socket says 172.21.15.173:43390) 2024-01-21T19:54:40.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.785+0000 7fdf8effd700 1 -- 172.21.15.173:0/587121028 learned_addr learned my addr 172.21.15.173:0/587121028 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:54:40.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.785+0000 7fdf8effd700 1 -- 172.21.15.173:0/587121028 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf80005040 con 0x7fdf900f5410 2024-01-21T19:54:40.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.785+0000 7fdf8effd700 1 --2- 172.21.15.173:0/587121028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf900f5410 0x7fdf90106700 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fdf80014000 tx=0x7fdf8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:40.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.785+0000 7fdf777fe700 1 -- 172.21.15.173:0/587121028 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf800095f0 con 0x7fdf900f5410 2024-01-21T19:54:40.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.786+0000 7fdf9599a700 1 -- 172.21.15.173:0/587121028 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf90106df0 con 0x7fdf900f5410 2024-01-21T19:54:40.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.786+0000 7fdf777fe700 1 -- 172.21.15.173:0/587121028 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf80007e70 con 0x7fdf900f5410 2024-01-21T19:54:40.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.786+0000 7fdf777fe700 1 -- 172.21.15.173:0/587121028 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf80014ea0 con 0x7fdf900f5410 2024-01-21T19:54:40.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.786+0000 7fdf9599a700 1 -- 172.21.15.173:0/587121028 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf90107280 con 0x7fdf900f5410 2024-01-21T19:54:40.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.786+0000 7fdf777fe700 1 -- 172.21.15.173:0/587121028 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdf80025070 con 0x7fdf900f5410 2024-01-21T19:54:40.779 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.786+0000 7fdf9599a700 1 -- 172.21.15.173:0/587121028 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf90045ad0 con 0x7fdf900f5410 2024-01-21T19:54:40.779 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.786+0000 7fdf777fe700 1 --2- 172.21.15.173:0/587121028 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fdf7802ff60 0x7fdf78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:40.779 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.787+0000 7fdf777fe700 1 -- 172.21.15.173:0/587121028 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdf800409f0 con 0x7fdf900f5410 2024-01-21T19:54:40.779 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.787+0000 7fdf8e7fc700 1 --2- 172.21.15.173:0/587121028 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fdf7802ff60 0x7fdf78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:40.779 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.787+0000 7fdf8e7fc700 1 --2- 172.21.15.173:0/587121028 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fdf7802ff60 0x7fdf78032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fdf840096a0 tx=0x7fdf84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:40.782 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.791+0000 7fdf777fe700 1 -- 172.21.15.173:0/587121028 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdf8000f4e0 con 0x7fdf900f5410 2024-01-21T19:54:40.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.998+0000 7fdf9599a700 1 -- 172.21.15.173:0/587121028 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf900fae60 con 0x7fdf900f5410 2024-01-21T19:54:40.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:40.999+0000 7fdf777fe700 1 -- 172.21.15.173:0/587121028 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdf8000f300 con 0x7fdf900f5410 2024-01-21T19:54:40.990 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:54:40.990 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:54:40.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:41.000+0000 7fdf9599a700 1 -- 172.21.15.173:0/587121028 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fdf7802ff60 msgr2=0x7fdf78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:40.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:41.000+0000 7fdf9599a700 1 --2- 172.21.15.173:0/587121028 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fdf7802ff60 0x7fdf78032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fdf840096a0 tx=0x7fdf84006b40).stop 2024-01-21T19:54:40.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:41.000+0000 7fdf9599a700 1 -- 172.21.15.173:0/587121028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf900f5410 msgr2=0x7fdf90106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:40.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:41.000+0000 7fdf9599a700 1 --2- 172.21.15.173:0/587121028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf900f5410 0x7fdf90106700 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fdf80014000 tx=0x7fdf8000b040).stop 2024-01-21T19:54:40.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:41.000+0000 7fdf9599a700 1 -- 172.21.15.173:0/587121028 shutdown_connections 2024-01-21T19:54:40.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:41.000+0000 7fdf9599a700 1 --2- 172.21.15.173:0/587121028 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fdf7802ff60 0x7fdf78032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:40.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:41.000+0000 7fdf9599a700 1 --2- 172.21.15.173:0/587121028 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf900f5410 0x7fdf90106700 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:40.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:41.000+0000 7fdf9599a700 1 -- 172.21.15.173:0/587121028 >> 172.21.15.173:0/587121028 conn(0x7fdf900f0b70 msgr2=0x7fdf900f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:40.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:41.000+0000 7fdf9599a700 1 -- 172.21.15.173:0/587121028 shutdown_connections 2024-01-21T19:54:40.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:41.000+0000 7fdf9599a700 1 -- 172.21.15.173:0/587121028 wait complete. 2024-01-21T19:54:40.993 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:54:41.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:40 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:38.990733+0000 mgr.smithi169.espgue (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:42.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:41 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:40.999530+0000 mon.smithi169 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.173:0/587121028' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:42.387 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:42.387 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:54:43.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:40.991111+0000 mgr.smithi169.espgue (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:43.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.550+0000 7fde52ff0700 1 -- 172.21.15.173:0/1183144803 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde4c0f5410 msgr2=0x7fde4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:43.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.550+0000 7fde52ff0700 1 --2- 172.21.15.173:0/1183144803 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde4c0f5410 0x7fde4c0f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fde40009fb0 tx=0x7fde40009b30).stop 2024-01-21T19:54:43.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.550+0000 7fde52ff0700 1 -- 172.21.15.173:0/1183144803 shutdown_connections 2024-01-21T19:54:43.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.550+0000 7fde52ff0700 1 --2- 172.21.15.173:0/1183144803 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde4c0f5410 0x7fde4c0f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:43.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.550+0000 7fde52ff0700 1 -- 172.21.15.173:0/1183144803 >> 172.21.15.173:0/1183144803 conn(0x7fde4c0f0b70 msgr2=0x7fde4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:43.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.550+0000 7fde52ff0700 1 -- 172.21.15.173:0/1183144803 shutdown_connections 2024-01-21T19:54:43.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.550+0000 7fde52ff0700 1 -- 172.21.15.173:0/1183144803 wait complete. 2024-01-21T19:54:43.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.551+0000 7fde52ff0700 1 Processor -- start 2024-01-21T19:54:43.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.551+0000 7fde52ff0700 1 -- start start 2024-01-21T19:54:43.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.551+0000 7fde52ff0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde4c0f5410 0x7fde4c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:43.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.551+0000 7fde52ff0700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c060240 con 0x7fde4c0f5410 2024-01-21T19:54:43.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.551+0000 7fde50d8c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde4c0f5410 0x7fde4c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:43.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.551+0000 7fde50d8c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde4c0f5410 0x7fde4c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:55410/0 (socket says 172.21.15.173:55410) 2024-01-21T19:54:43.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.551+0000 7fde50d8c700 1 -- 172.21.15.173:0/4201061612 learned_addr learned my addr 172.21.15.173:0/4201061612 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:54:43.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.552+0000 7fde50d8c700 1 -- 172.21.15.173:0/4201061612 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde40005040 con 0x7fde4c0f5410 2024-01-21T19:54:43.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.552+0000 7fde50d8c700 1 --2- 172.21.15.173:0/4201061612 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde4c0f5410 0x7fde4c109430 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fde40013510 tx=0x7fde4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:43.545 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.552+0000 7fde497fa700 1 -- 172.21.15.173:0/4201061612 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde40009320 con 0x7fde4c0f5410 2024-01-21T19:54:43.545 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.552+0000 7fde52ff0700 1 -- 172.21.15.173:0/4201061612 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde4c060440 con 0x7fde4c0f5410 2024-01-21T19:54:43.545 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.552+0000 7fde52ff0700 1 -- 172.21.15.173:0/4201061612 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde4c109ae0 con 0x7fde4c0f5410 2024-01-21T19:54:43.546 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.552+0000 7fde497fa700 1 -- 172.21.15.173:0/4201061612 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde40013f00 con 0x7fde4c0f5410 2024-01-21T19:54:43.546 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.552+0000 7fde497fa700 1 -- 172.21.15.173:0/4201061612 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde40014d60 con 0x7fde4c0f5410 2024-01-21T19:54:43.546 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.553+0000 7fde497fa700 1 -- 172.21.15.173:0/4201061612 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fde400263a0 con 0x7fde4c0f5410 2024-01-21T19:54:43.546 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.553+0000 7fde497fa700 1 --2- 172.21.15.173:0/4201061612 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fde3c02ff60 0x7fde3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:43.546 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.553+0000 7fde52ff0700 1 -- 172.21.15.173:0/4201061612 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde4c045ad0 con 0x7fde4c0f5410 2024-01-21T19:54:43.546 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.553+0000 7fde497fa700 1 -- 172.21.15.173:0/4201061612 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fde400412e0 con 0x7fde4c0f5410 2024-01-21T19:54:43.546 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.553+0000 7fde4bfff700 1 --2- 172.21.15.173:0/4201061612 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fde3c02ff60 0x7fde3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:43.546 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.554+0000 7fde4bfff700 1 --2- 172.21.15.173:0/4201061612 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fde3c02ff60 0x7fde3c032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fde380096a0 tx=0x7fde38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:43.549 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.558+0000 7fde497fa700 1 -- 172.21.15.173:0/4201061612 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fde40012330 con 0x7fde4c0f5410 2024-01-21T19:54:43.756 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.764+0000 7fde52ff0700 1 -- 172.21.15.173:0/4201061612 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde4c0faf50 con 0x7fde4c0f5410 2024-01-21T19:54:43.756 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.765+0000 7fde497fa700 1 -- 172.21.15.173:0/4201061612 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fde40014420 con 0x7fde4c0f5410 2024-01-21T19:54:43.757 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:54:43.757 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:54:43.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.767+0000 7fde52ff0700 1 -- 172.21.15.173:0/4201061612 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fde3c02ff60 msgr2=0x7fde3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:43.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.767+0000 7fde52ff0700 1 --2- 172.21.15.173:0/4201061612 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fde3c02ff60 0x7fde3c032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fde380096a0 tx=0x7fde38006b40).stop 2024-01-21T19:54:43.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.767+0000 7fde52ff0700 1 -- 172.21.15.173:0/4201061612 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde4c0f5410 msgr2=0x7fde4c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:43.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.767+0000 7fde52ff0700 1 --2- 172.21.15.173:0/4201061612 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde4c0f5410 0x7fde4c109430 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fde40013510 tx=0x7fde4000b040).stop 2024-01-21T19:54:43.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.767+0000 7fde52ff0700 1 -- 172.21.15.173:0/4201061612 shutdown_connections 2024-01-21T19:54:43.759 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.767+0000 7fde52ff0700 1 --2- 172.21.15.173:0/4201061612 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fde3c02ff60 0x7fde3c032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:43.759 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.767+0000 7fde52ff0700 1 --2- 172.21.15.173:0/4201061612 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fde4c0f5410 0x7fde4c109430 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:43.759 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.767+0000 7fde52ff0700 1 -- 172.21.15.173:0/4201061612 >> 172.21.15.173:0/4201061612 conn(0x7fde4c0f0b70 msgr2=0x7fde4c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:43.759 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.767+0000 7fde52ff0700 1 -- 172.21.15.173:0/4201061612 shutdown_connections 2024-01-21T19:54:43.759 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:43.767+0000 7fde52ff0700 1 -- 172.21.15.173:0/4201061612 wait complete. 2024-01-21T19:54:43.759 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:54:45.144 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:45.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:54:45.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:44 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:42.991496+0000 mgr.smithi169.espgue (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:45.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:44 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:43.765717+0000 mon.smithi169 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.173:0/4201061612' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:46.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.364+0000 7fa8509bc700 1 -- 172.21.15.173:0/546545882 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa84c0f62a0 msgr2=0x7fa84c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:46.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.364+0000 7fa8509bc700 1 --2- 172.21.15.173:0/546545882 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa84c0f62a0 0x7fa84c0f66c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fa83c009fb0 tx=0x7fa83c009b30).stop 2024-01-21T19:54:46.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.365+0000 7fa8509bc700 1 -- 172.21.15.173:0/546545882 shutdown_connections 2024-01-21T19:54:46.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.365+0000 7fa8509bc700 1 --2- 172.21.15.173:0/546545882 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa84c0f62a0 0x7fa84c0f66c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:46.357 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.365+0000 7fa8509bc700 1 -- 172.21.15.173:0/546545882 >> 172.21.15.173:0/546545882 conn(0x7fa84c0f1aa0 msgr2=0x7fa84c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:46.357 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.365+0000 7fa8509bc700 1 -- 172.21.15.173:0/546545882 shutdown_connections 2024-01-21T19:54:46.357 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.365+0000 7fa8509bc700 1 -- 172.21.15.173:0/546545882 wait complete. 2024-01-21T19:54:46.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.365+0000 7fa8509bc700 1 Processor -- start 2024-01-21T19:54:46.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.365+0000 7fa8509bc700 1 -- start start 2024-01-21T19:54:46.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.365+0000 7fa8509bc700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa84c0f62a0 0x7fa84c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:46.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.365+0000 7fa8509bc700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c1069a0 con 0x7fa84c0f62a0 2024-01-21T19:54:46.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.366+0000 7fa84ad9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa84c0f62a0 0x7fa84c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:46.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.366+0000 7fa84ad9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa84c0f62a0 0x7fa84c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:55426/0 (socket says 172.21.15.173:55426) 2024-01-21T19:54:46.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.366+0000 7fa84ad9d700 1 -- 172.21.15.173:0/3761893073 learned_addr learned my addr 172.21.15.173:0/3761893073 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:54:46.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.366+0000 7fa84ad9d700 1 -- 172.21.15.173:0/3761893073 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83c005040 con 0x7fa84c0f62a0 2024-01-21T19:54:46.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.366+0000 7fa84ad9d700 1 --2- 172.21.15.173:0/3761893073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa84c0f62a0 0x7fa84c1064b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fa83c013510 tx=0x7fa83c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:46.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.366+0000 7fa8337fe700 1 -- 172.21.15.173:0/3761893073 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa83c009320 con 0x7fa84c0f62a0 2024-01-21T19:54:46.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.366+0000 7fa8509bc700 1 -- 172.21.15.173:0/3761893073 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84c106ba0 con 0x7fa84c0f62a0 2024-01-21T19:54:46.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.366+0000 7fa8337fe700 1 -- 172.21.15.173:0/3761893073 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa83c013f00 con 0x7fa84c0f62a0 2024-01-21T19:54:46.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.366+0000 7fa8337fe700 1 -- 172.21.15.173:0/3761893073 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa83c014dc0 con 0x7fa84c0f62a0 2024-01-21T19:54:46.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.366+0000 7fa8509bc700 1 -- 172.21.15.173:0/3761893073 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84c107030 con 0x7fa84c0f62a0 2024-01-21T19:54:46.360 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.367+0000 7fa8337fe700 1 -- 172.21.15.173:0/3761893073 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa83c014420 con 0x7fa84c0f62a0 2024-01-21T19:54:46.360 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.367+0000 7fa8509bc700 1 -- 172.21.15.173:0/3761893073 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa84c045ad0 con 0x7fa84c0f62a0 2024-01-21T19:54:46.360 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.367+0000 7fa8337fe700 1 --2- 172.21.15.173:0/3761893073 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa83402ff60 0x7fa834032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:46.360 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.367+0000 7fa8337fe700 1 -- 172.21.15.173:0/3761893073 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa83c0410b0 con 0x7fa84c0f62a0 2024-01-21T19:54:46.360 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.367+0000 7fa84a59c700 1 --2- 172.21.15.173:0/3761893073 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa83402ff60 0x7fa834032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:46.360 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.368+0000 7fa84a59c700 1 --2- 172.21.15.173:0/3761893073 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa83402ff60 0x7fa834032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fa8400096a0 tx=0x7fa840006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:46.363 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.372+0000 7fa8337fe700 1 -- 172.21.15.173:0/3761893073 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa83c0095f0 con 0x7fa84c0f62a0 2024-01-21T19:54:46.568 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.576+0000 7fa8509bc700 1 -- 172.21.15.173:0/3761893073 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa84c107ee0 con 0x7fa84c0f62a0 2024-01-21T19:54:46.568 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.577+0000 7fa8337fe700 1 -- 172.21.15.173:0/3761893073 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa83c01cb10 con 0x7fa84c0f62a0 2024-01-21T19:54:46.569 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:54:46.569 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:54:46.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.578+0000 7fa8509bc700 1 -- 172.21.15.173:0/3761893073 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa83402ff60 msgr2=0x7fa834032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:46.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.578+0000 7fa8509bc700 1 --2- 172.21.15.173:0/3761893073 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa83402ff60 0x7fa834032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fa8400096a0 tx=0x7fa840006b40).stop 2024-01-21T19:54:46.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.578+0000 7fa8509bc700 1 -- 172.21.15.173:0/3761893073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa84c0f62a0 msgr2=0x7fa84c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:46.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.578+0000 7fa8509bc700 1 --2- 172.21.15.173:0/3761893073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa84c0f62a0 0x7fa84c1064b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fa83c013510 tx=0x7fa83c00b040).stop 2024-01-21T19:54:46.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.578+0000 7fa8509bc700 1 -- 172.21.15.173:0/3761893073 shutdown_connections 2024-01-21T19:54:46.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.578+0000 7fa8509bc700 1 --2- 172.21.15.173:0/3761893073 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa83402ff60 0x7fa834032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:46.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.578+0000 7fa8509bc700 1 --2- 172.21.15.173:0/3761893073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa84c0f62a0 0x7fa84c1064b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:46.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.578+0000 7fa8509bc700 1 -- 172.21.15.173:0/3761893073 >> 172.21.15.173:0/3761893073 conn(0x7fa84c0f1aa0 msgr2=0x7fa84c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:46.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.579+0000 7fa8509bc700 1 -- 172.21.15.173:0/3761893073 shutdown_connections 2024-01-21T19:54:46.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:46.579+0000 7fa8509bc700 1 -- 172.21.15.173:0/3761893073 wait complete. 2024-01-21T19:54:46.571 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:54:47.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:46 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:44.991801+0000 mgr.smithi169.espgue (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:47.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:46 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:46.577669+0000 mon.smithi169 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.173:0/3761893073' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:47.951 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:47.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:54:49.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:46.992081+0000 mgr.smithi169.espgue (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:49.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.185+0000 7fe779b3a700 1 -- 172.21.15.173:0/2050200865 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7740f5410 msgr2=0x7fe7740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:49.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.185+0000 7fe779b3a700 1 --2- 172.21.15.173:0/2050200865 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7740f5410 0x7fe7740f5830 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fe75c009fb0 tx=0x7fe75c009b30).stop 2024-01-21T19:54:49.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.185+0000 7fe779b3a700 1 -- 172.21.15.173:0/2050200865 shutdown_connections 2024-01-21T19:54:49.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.185+0000 7fe779b3a700 1 --2- 172.21.15.173:0/2050200865 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7740f5410 0x7fe7740f5830 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:49.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.185+0000 7fe779b3a700 1 -- 172.21.15.173:0/2050200865 >> 172.21.15.173:0/2050200865 conn(0x7fe7740f0b70 msgr2=0x7fe7740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:49.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.185+0000 7fe779b3a700 1 -- 172.21.15.173:0/2050200865 shutdown_connections 2024-01-21T19:54:49.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.185+0000 7fe779b3a700 1 -- 172.21.15.173:0/2050200865 wait complete. 2024-01-21T19:54:49.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.186+0000 7fe779b3a700 1 Processor -- start 2024-01-21T19:54:49.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.186+0000 7fe779b3a700 1 -- start start 2024-01-21T19:54:49.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.186+0000 7fe779b3a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7740f5410 0x7fe774109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:49.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.186+0000 7fe779b3a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe774060220 con 0x7fe7740f5410 2024-01-21T19:54:49.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.186+0000 7fe7737fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7740f5410 0x7fe774109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:49.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.186+0000 7fe7737fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7740f5410 0x7fe774109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:55450/0 (socket says 172.21.15.173:55450) 2024-01-21T19:54:49.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.186+0000 7fe7737fe700 1 -- 172.21.15.173:0/1876092327 learned_addr learned my addr 172.21.15.173:0/1876092327 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:54:49.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.187+0000 7fe7737fe700 1 -- 172.21.15.173:0/1876092327 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75c005040 con 0x7fe7740f5410 2024-01-21T19:54:49.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.187+0000 7fe7737fe700 1 --2- 172.21.15.173:0/1876092327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7740f5410 0x7fe774109450 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fe75c014000 tx=0x7fe75c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:49.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.187+0000 7fe76bfff700 1 -- 172.21.15.173:0/1876092327 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe75c0095f0 con 0x7fe7740f5410 2024-01-21T19:54:49.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.187+0000 7fe779b3a700 1 -- 172.21.15.173:0/1876092327 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe774060420 con 0x7fe7740f5410 2024-01-21T19:54:49.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.187+0000 7fe76bfff700 1 -- 172.21.15.173:0/1876092327 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe75c007e70 con 0x7fe7740f5410 2024-01-21T19:54:49.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.187+0000 7fe76bfff700 1 -- 172.21.15.173:0/1876092327 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe75c014ea0 con 0x7fe7740f5410 2024-01-21T19:54:49.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.187+0000 7fe779b3a700 1 -- 172.21.15.173:0/1876092327 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe774109b00 con 0x7fe7740f5410 2024-01-21T19:54:49.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.188+0000 7fe76bfff700 1 -- 172.21.15.173:0/1876092327 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe75c014690 con 0x7fe7740f5410 2024-01-21T19:54:49.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.188+0000 7fe779b3a700 1 -- 172.21.15.173:0/1876092327 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe774059100 con 0x7fe7740f5410 2024-01-21T19:54:49.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.188+0000 7fe76bfff700 1 --2- 172.21.15.173:0/1876092327 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe76002ffb0 0x7fe760032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:49.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.188+0000 7fe76bfff700 1 -- 172.21.15.173:0/1876092327 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe75c0408f0 con 0x7fe7740f5410 2024-01-21T19:54:49.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.188+0000 7fe772ffd700 1 --2- 172.21.15.173:0/1876092327 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe76002ffb0 0x7fe760032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:49.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.189+0000 7fe772ffd700 1 --2- 172.21.15.173:0/1876092327 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe76002ffb0 0x7fe760032470 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fe7640096a0 tx=0x7fe764006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:49.184 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.193+0000 7fe76bfff700 1 -- 172.21.15.173:0/1876092327 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe75c00f4e0 con 0x7fe7740f5410 2024-01-21T19:54:49.391 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.400+0000 7fe779b3a700 1 -- 172.21.15.173:0/1876092327 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe774045ad0 con 0x7fe7740f5410 2024-01-21T19:54:49.392 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.400+0000 7fe76bfff700 1 -- 172.21.15.173:0/1876092327 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe75c00f300 con 0x7fe7740f5410 2024-01-21T19:54:49.392 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:54:49.392 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:54:49.394 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.402+0000 7fe779b3a700 1 -- 172.21.15.173:0/1876092327 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe76002ffb0 msgr2=0x7fe760032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:49.394 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.402+0000 7fe779b3a700 1 --2- 172.21.15.173:0/1876092327 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe76002ffb0 0x7fe760032470 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fe7640096a0 tx=0x7fe764006b40).stop 2024-01-21T19:54:49.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.402+0000 7fe779b3a700 1 -- 172.21.15.173:0/1876092327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7740f5410 msgr2=0x7fe774109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:49.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.402+0000 7fe779b3a700 1 --2- 172.21.15.173:0/1876092327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7740f5410 0x7fe774109450 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fe75c014000 tx=0x7fe75c00b040).stop 2024-01-21T19:54:49.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.402+0000 7fe779b3a700 1 -- 172.21.15.173:0/1876092327 shutdown_connections 2024-01-21T19:54:49.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.402+0000 7fe779b3a700 1 --2- 172.21.15.173:0/1876092327 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe76002ffb0 0x7fe760032470 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:49.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.402+0000 7fe779b3a700 1 --2- 172.21.15.173:0/1876092327 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe7740f5410 0x7fe774109450 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:49.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.402+0000 7fe779b3a700 1 -- 172.21.15.173:0/1876092327 >> 172.21.15.173:0/1876092327 conn(0x7fe7740f0b70 msgr2=0x7fe7740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:49.396 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.402+0000 7fe779b3a700 1 -- 172.21.15.173:0/1876092327 shutdown_connections 2024-01-21T19:54:49.396 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:49.402+0000 7fe779b3a700 1 -- 172.21.15.173:0/1876092327 wait complete. 2024-01-21T19:54:49.396 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:54:50.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:49 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:49.401256+0000 mon.smithi169 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.173:0/1876092327' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:50.781 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:50.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:54:51.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:50 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:48.992317+0000 mgr.smithi169.espgue (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:52.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.028+0000 7f5e9a1c2700 1 -- 172.21.15.173:0/1229039703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e940f5410 msgr2=0x7f5e940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:52.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.028+0000 7f5e9a1c2700 1 --2- 172.21.15.173:0/1229039703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e940f5410 0x7f5e940f5830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f5e7c009fb0 tx=0x7f5e7c009b30).stop 2024-01-21T19:54:52.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.028+0000 7f5e9a1c2700 1 -- 172.21.15.173:0/1229039703 shutdown_connections 2024-01-21T19:54:52.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.028+0000 7f5e9a1c2700 1 --2- 172.21.15.173:0/1229039703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e940f5410 0x7f5e940f5830 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:52.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.028+0000 7f5e9a1c2700 1 -- 172.21.15.173:0/1229039703 >> 172.21.15.173:0/1229039703 conn(0x7f5e940f0b70 msgr2=0x7f5e940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:52.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.028+0000 7f5e9a1c2700 1 -- 172.21.15.173:0/1229039703 shutdown_connections 2024-01-21T19:54:52.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.028+0000 7f5e9a1c2700 1 -- 172.21.15.173:0/1229039703 wait complete. 2024-01-21T19:54:52.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.029+0000 7f5e9a1c2700 1 Processor -- start 2024-01-21T19:54:52.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.029+0000 7f5e9a1c2700 1 -- start start 2024-01-21T19:54:52.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.029+0000 7f5e9a1c2700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e940f5410 0x7f5e94109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:52.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.029+0000 7f5e9a1c2700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e94060220 con 0x7f5e940f5410 2024-01-21T19:54:52.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.029+0000 7f5e937fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e940f5410 0x7f5e94109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:52.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.029+0000 7f5e937fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e940f5410 0x7f5e94109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:55464/0 (socket says 172.21.15.173:55464) 2024-01-21T19:54:52.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.029+0000 7f5e937fe700 1 -- 172.21.15.173:0/2815799589 learned_addr learned my addr 172.21.15.173:0/2815799589 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:54:52.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.030+0000 7f5e937fe700 1 -- 172.21.15.173:0/2815799589 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7c005040 con 0x7f5e940f5410 2024-01-21T19:54:52.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.030+0000 7f5e937fe700 1 --2- 172.21.15.173:0/2815799589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e940f5410 0x7f5e94109450 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f5e7c013510 tx=0x7f5e7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:52.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.030+0000 7f5e8bfff700 1 -- 172.21.15.173:0/2815799589 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e7c009320 con 0x7f5e940f5410 2024-01-21T19:54:52.023 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.030+0000 7f5e9a1c2700 1 -- 172.21.15.173:0/2815799589 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e94060420 con 0x7f5e940f5410 2024-01-21T19:54:52.023 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.030+0000 7f5e8bfff700 1 -- 172.21.15.173:0/2815799589 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e7c013f00 con 0x7f5e940f5410 2024-01-21T19:54:52.023 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.030+0000 7f5e8bfff700 1 -- 172.21.15.173:0/2815799589 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e7c014d90 con 0x7f5e940f5410 2024-01-21T19:54:52.023 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.030+0000 7f5e9a1c2700 1 -- 172.21.15.173:0/2815799589 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e94109b00 con 0x7f5e940f5410 2024-01-21T19:54:52.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.031+0000 7f5e8bfff700 1 -- 172.21.15.173:0/2815799589 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e7c0263a0 con 0x7f5e940f5410 2024-01-21T19:54:52.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.031+0000 7f5e8bfff700 1 --2- 172.21.15.173:0/2815799589 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5e8002ff60 0x7f5e80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:52.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.031+0000 7f5e8bfff700 1 -- 172.21.15.173:0/2815799589 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e7c0410c0 con 0x7f5e940f5410 2024-01-21T19:54:52.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.031+0000 7f5e9a1c2700 1 -- 172.21.15.173:0/2815799589 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e94059100 con 0x7f5e940f5410 2024-01-21T19:54:52.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.031+0000 7f5e92ffd700 1 --2- 172.21.15.173:0/2815799589 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5e8002ff60 0x7f5e80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:52.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.031+0000 7f5e92ffd700 1 --2- 172.21.15.173:0/2815799589 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5e8002ff60 0x7f5e80032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f5e840096a0 tx=0x7f5e84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:52.027 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.036+0000 7f5e8bfff700 1 -- 172.21.15.173:0/2815799589 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e7c012920 con 0x7f5e940f5410 2024-01-21T19:54:52.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.238+0000 7f5e9a1c2700 1 -- 172.21.15.173:0/2815799589 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e9410a1e0 con 0x7f5e940f5410 2024-01-21T19:54:52.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.239+0000 7f5e8bfff700 1 -- 172.21.15.173:0/2815799589 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5e7c02b020 con 0x7f5e940f5410 2024-01-21T19:54:52.230 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:54:52.231 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:54:52.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.240+0000 7f5e9a1c2700 1 -- 172.21.15.173:0/2815799589 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5e8002ff60 msgr2=0x7f5e80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:52.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.240+0000 7f5e9a1c2700 1 --2- 172.21.15.173:0/2815799589 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5e8002ff60 0x7f5e80032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f5e840096a0 tx=0x7f5e84006b40).stop 2024-01-21T19:54:52.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.240+0000 7f5e9a1c2700 1 -- 172.21.15.173:0/2815799589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e940f5410 msgr2=0x7f5e94109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:52.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.240+0000 7f5e9a1c2700 1 --2- 172.21.15.173:0/2815799589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e940f5410 0x7f5e94109450 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f5e7c013510 tx=0x7f5e7c00b040).stop 2024-01-21T19:54:52.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.240+0000 7f5e9a1c2700 1 -- 172.21.15.173:0/2815799589 shutdown_connections 2024-01-21T19:54:52.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.240+0000 7f5e9a1c2700 1 --2- 172.21.15.173:0/2815799589 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5e8002ff60 0x7f5e80032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:52.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.240+0000 7f5e9a1c2700 1 --2- 172.21.15.173:0/2815799589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e940f5410 0x7f5e94109450 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:52.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.240+0000 7f5e9a1c2700 1 -- 172.21.15.173:0/2815799589 >> 172.21.15.173:0/2815799589 conn(0x7f5e940f0b70 msgr2=0x7f5e940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:52.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.240+0000 7f5e9a1c2700 1 -- 172.21.15.173:0/2815799589 shutdown_connections 2024-01-21T19:54:52.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:52.241+0000 7f5e9a1c2700 1 -- 172.21.15.173:0/2815799589 wait complete. 2024-01-21T19:54:52.233 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:54:53.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:52 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:50.992688+0000 mgr.smithi169.espgue (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:53.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:52 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:52.239584+0000 mon.smithi169 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.173:0/2815799589' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:53.628 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:53.628 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:54:54.833 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.840+0000 7fb59dcaf700 1 -- 172.21.15.173:0/131852897 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5980f62a0 msgr2=0x7fb5980f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:54.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.840+0000 7fb59dcaf700 1 --2- 172.21.15.173:0/131852897 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5980f62a0 0x7fb5980f66c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fb58c009fb0 tx=0x7fb58c009b30).stop 2024-01-21T19:54:54.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.840+0000 7fb59dcaf700 1 -- 172.21.15.173:0/131852897 shutdown_connections 2024-01-21T19:54:54.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.840+0000 7fb59dcaf700 1 --2- 172.21.15.173:0/131852897 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5980f62a0 0x7fb5980f66c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:54.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.840+0000 7fb59dcaf700 1 -- 172.21.15.173:0/131852897 >> 172.21.15.173:0/131852897 conn(0x7fb5980f1aa0 msgr2=0x7fb5980f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:54.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.840+0000 7fb59dcaf700 1 -- 172.21.15.173:0/131852897 shutdown_connections 2024-01-21T19:54:54.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.840+0000 7fb59dcaf700 1 -- 172.21.15.173:0/131852897 wait complete. 2024-01-21T19:54:54.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.840+0000 7fb59dcaf700 1 Processor -- start 2024-01-21T19:54:54.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.840+0000 7fb59dcaf700 1 -- start start 2024-01-21T19:54:54.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.841+0000 7fb59dcaf700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5980f62a0 0x7fb5981064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:54.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.841+0000 7fb59dcaf700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5981069a0 con 0x7fb5980f62a0 2024-01-21T19:54:54.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.841+0000 7fb59ccad700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5980f62a0 0x7fb5981064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:54.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.841+0000 7fb59ccad700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5980f62a0 0x7fb5981064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:39972/0 (socket says 172.21.15.173:39972) 2024-01-21T19:54:54.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.841+0000 7fb59ccad700 1 -- 172.21.15.173:0/1974795110 learned_addr learned my addr 172.21.15.173:0/1974795110 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:54:54.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.841+0000 7fb59ccad700 1 -- 172.21.15.173:0/1974795110 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb58c005040 con 0x7fb5980f62a0 2024-01-21T19:54:54.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.841+0000 7fb59ccad700 1 --2- 172.21.15.173:0/1974795110 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5980f62a0 0x7fb5981064b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fb58c013510 tx=0x7fb58c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:54.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.842+0000 7fb5957fa700 1 -- 172.21.15.173:0/1974795110 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb58c009320 con 0x7fb5980f62a0 2024-01-21T19:54:54.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.842+0000 7fb59dcaf700 1 -- 172.21.15.173:0/1974795110 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb598106ba0 con 0x7fb5980f62a0 2024-01-21T19:54:54.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.842+0000 7fb5957fa700 1 -- 172.21.15.173:0/1974795110 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb58c013f00 con 0x7fb5980f62a0 2024-01-21T19:54:54.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.842+0000 7fb5957fa700 1 -- 172.21.15.173:0/1974795110 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb58c014dc0 con 0x7fb5980f62a0 2024-01-21T19:54:54.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.842+0000 7fb59dcaf700 1 -- 172.21.15.173:0/1974795110 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb598107030 con 0x7fb5980f62a0 2024-01-21T19:54:54.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.842+0000 7fb5957fa700 1 -- 172.21.15.173:0/1974795110 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb58c01d3d0 con 0x7fb5980f62a0 2024-01-21T19:54:54.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.843+0000 7fb59dcaf700 1 -- 172.21.15.173:0/1974795110 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb598045ad0 con 0x7fb5980f62a0 2024-01-21T19:54:54.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.843+0000 7fb5957fa700 1 --2- 172.21.15.173:0/1974795110 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb58802ff60 0x7fb588032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:54.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.843+0000 7fb5957fa700 1 -- 172.21.15.173:0/1974795110 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb58c0410e0 con 0x7fb5980f62a0 2024-01-21T19:54:54.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.843+0000 7fb597fff700 1 --2- 172.21.15.173:0/1974795110 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb58802ff60 0x7fb588032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:54.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.843+0000 7fb597fff700 1 --2- 172.21.15.173:0/1974795110 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb58802ff60 0x7fb588032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fb5840096a0 tx=0x7fb584006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:54.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:54.847+0000 7fb5957fa700 1 -- 172.21.15.173:0/1974795110 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb58c0095f0 con 0x7fb5980f62a0 2024-01-21T19:54:55.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:55.053+0000 7fb59dcaf700 1 -- 172.21.15.173:0/1974795110 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb598107ee0 con 0x7fb5980f62a0 2024-01-21T19:54:55.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:55.053+0000 7fb5957fa700 1 -- 172.21.15.173:0/1974795110 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb58c01cb10 con 0x7fb5980f62a0 2024-01-21T19:54:55.045 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:54:55.045 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:54:55.047 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:55.055+0000 7fb59dcaf700 1 -- 172.21.15.173:0/1974795110 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb58802ff60 msgr2=0x7fb588032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:55.047 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:55.055+0000 7fb59dcaf700 1 --2- 172.21.15.173:0/1974795110 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb58802ff60 0x7fb588032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fb5840096a0 tx=0x7fb584006b40).stop 2024-01-21T19:54:55.047 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:55.055+0000 7fb59dcaf700 1 -- 172.21.15.173:0/1974795110 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5980f62a0 msgr2=0x7fb5981064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:55.047 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:55.055+0000 7fb59dcaf700 1 --2- 172.21.15.173:0/1974795110 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5980f62a0 0x7fb5981064b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fb58c013510 tx=0x7fb58c00b040).stop 2024-01-21T19:54:55.047 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:55.055+0000 7fb59dcaf700 1 -- 172.21.15.173:0/1974795110 shutdown_connections 2024-01-21T19:54:55.047 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:55.055+0000 7fb59dcaf700 1 --2- 172.21.15.173:0/1974795110 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fb58802ff60 0x7fb588032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:55.047 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:55.055+0000 7fb59dcaf700 1 --2- 172.21.15.173:0/1974795110 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb5980f62a0 0x7fb5981064b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:55.047 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:55.055+0000 7fb59dcaf700 1 -- 172.21.15.173:0/1974795110 >> 172.21.15.173:0/1974795110 conn(0x7fb5980f1aa0 msgr2=0x7fb5980f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:55.047 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:55.055+0000 7fb59dcaf700 1 -- 172.21.15.173:0/1974795110 shutdown_connections 2024-01-21T19:54:55.047 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:55.055+0000 7fb59dcaf700 1 -- 172.21.15.173:0/1974795110 wait complete. 2024-01-21T19:54:55.048 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:54:55.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:54 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:52.993150+0000 mgr.smithi169.espgue (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:56.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:55 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:55.054285+0000 mon.smithi169 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.173:0/1974795110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:56.440 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:56.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:54:57.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:56 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:54.993502+0000 mgr.smithi169.espgue (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:57.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.637+0000 7f3edc09c700 1 -- 172.21.15.173:0/3428259435 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ed40f6510 msgr2=0x7f3ed40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:57.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.637+0000 7f3edc09c700 1 --2- 172.21.15.173:0/3428259435 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ed40f6510 0x7f3ed40f6930 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f3ec4009fb0 tx=0x7f3ec4009b30).stop 2024-01-21T19:54:57.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.637+0000 7f3edc09c700 1 -- 172.21.15.173:0/3428259435 shutdown_connections 2024-01-21T19:54:57.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.637+0000 7f3edc09c700 1 --2- 172.21.15.173:0/3428259435 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ed40f6510 0x7f3ed40f6930 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:57.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.637+0000 7f3edc09c700 1 -- 172.21.15.173:0/3428259435 >> 172.21.15.173:0/3428259435 conn(0x7f3ed40f1c70 msgr2=0x7f3ed40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:57.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.637+0000 7f3edc09c700 1 -- 172.21.15.173:0/3428259435 shutdown_connections 2024-01-21T19:54:57.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.637+0000 7f3edc09c700 1 -- 172.21.15.173:0/3428259435 wait complete. 2024-01-21T19:54:57.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.638+0000 7f3edc09c700 1 Processor -- start 2024-01-21T19:54:57.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.638+0000 7f3edc09c700 1 -- start start 2024-01-21T19:54:57.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.638+0000 7f3edc09c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ed40f6510 0x7f3ed4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:57.631 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.638+0000 7f3edc09c700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed4106c20 con 0x7f3ed40f6510 2024-01-21T19:54:57.631 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.638+0000 7f3ed9e38700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ed40f6510 0x7f3ed4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:57.631 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.639+0000 7f3ed9e38700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ed40f6510 0x7f3ed4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:39982/0 (socket says 172.21.15.173:39982) 2024-01-21T19:54:57.631 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.639+0000 7f3ed9e38700 1 -- 172.21.15.173:0/848866677 learned_addr learned my addr 172.21.15.173:0/848866677 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:54:57.632 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.639+0000 7f3ed9e38700 1 -- 172.21.15.173:0/848866677 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec4005040 con 0x7f3ed40f6510 2024-01-21T19:54:57.632 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.639+0000 7f3ed9e38700 1 --2- 172.21.15.173:0/848866677 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ed40f6510 0x7f3ed4106730 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f3ec4013510 tx=0x7f3ec400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:57.632 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.639+0000 7f3eca7fc700 1 -- 172.21.15.173:0/848866677 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ec4009320 con 0x7f3ed40f6510 2024-01-21T19:54:57.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.639+0000 7f3edc09c700 1 -- 172.21.15.173:0/848866677 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed4106e20 con 0x7f3ed40f6510 2024-01-21T19:54:57.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.639+0000 7f3eca7fc700 1 -- 172.21.15.173:0/848866677 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ec4013f00 con 0x7f3ed40f6510 2024-01-21T19:54:57.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.639+0000 7f3eca7fc700 1 -- 172.21.15.173:0/848866677 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ec4014d90 con 0x7f3ed40f6510 2024-01-21T19:54:57.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.639+0000 7f3edc09c700 1 -- 172.21.15.173:0/848866677 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed41072b0 con 0x7f3ed40f6510 2024-01-21T19:54:57.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.640+0000 7f3eca7fc700 1 -- 172.21.15.173:0/848866677 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ec40095f0 con 0x7f3ed40f6510 2024-01-21T19:54:57.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.640+0000 7f3edc09c700 1 -- 172.21.15.173:0/848866677 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ed4045ad0 con 0x7f3ed40f6510 2024-01-21T19:54:57.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.640+0000 7f3eca7fc700 1 --2- 172.21.15.173:0/848866677 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3ec002ff60 0x7f3ec0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:54:57.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.640+0000 7f3eca7fc700 1 -- 172.21.15.173:0/848866677 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ec4040fe0 con 0x7f3ed40f6510 2024-01-21T19:54:57.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.640+0000 7f3ed9637700 1 --2- 172.21.15.173:0/848866677 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3ec002ff60 0x7f3ec0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:54:57.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.641+0000 7f3ed9637700 1 --2- 172.21.15.173:0/848866677 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3ec002ff60 0x7f3ec0032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f3ed00096a0 tx=0x7f3ed0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:54:57.636 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.645+0000 7f3eca7fc700 1 -- 172.21.15.173:0/848866677 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ec401d7f0 con 0x7f3ed40f6510 2024-01-21T19:54:57.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.850+0000 7f3edc09c700 1 -- 172.21.15.173:0/848866677 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ed40fbf60 con 0x7f3ed40f6510 2024-01-21T19:54:57.843 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.851+0000 7f3eca7fc700 1 -- 172.21.15.173:0/848866677 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3ec4010230 con 0x7f3ed40f6510 2024-01-21T19:54:57.843 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:54:57.843 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:54:57.844 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.853+0000 7f3edc09c700 1 -- 172.21.15.173:0/848866677 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3ec002ff60 msgr2=0x7f3ec0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.853+0000 7f3edc09c700 1 --2- 172.21.15.173:0/848866677 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3ec002ff60 0x7f3ec0032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f3ed00096a0 tx=0x7f3ed0006b40).stop 2024-01-21T19:54:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.853+0000 7f3edc09c700 1 -- 172.21.15.173:0/848866677 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ed40f6510 msgr2=0x7f3ed4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:54:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.853+0000 7f3edc09c700 1 --2- 172.21.15.173:0/848866677 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ed40f6510 0x7f3ed4106730 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f3ec4013510 tx=0x7f3ec400b040).stop 2024-01-21T19:54:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.853+0000 7f3edc09c700 1 -- 172.21.15.173:0/848866677 shutdown_connections 2024-01-21T19:54:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.853+0000 7f3edc09c700 1 --2- 172.21.15.173:0/848866677 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3ec002ff60 0x7f3ec0032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.853+0000 7f3edc09c700 1 --2- 172.21.15.173:0/848866677 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ed40f6510 0x7f3ed4106730 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:54:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.853+0000 7f3edc09c700 1 -- 172.21.15.173:0/848866677 >> 172.21.15.173:0/848866677 conn(0x7f3ed40f1c70 msgr2=0x7f3ed40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:54:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.853+0000 7f3edc09c700 1 -- 172.21.15.173:0/848866677 shutdown_connections 2024-01-21T19:54:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:54:57.853+0000 7f3edc09c700 1 -- 172.21.15.173:0/848866677 wait complete. 2024-01-21T19:54:57.846 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:54:59.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:56.993864+0000 mgr.smithi169.espgue (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:54:59.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:54:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:54:57.851958+0000 mon.smithi169 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.173:0/848866677' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:54:59.244 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:54:59.244 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:00.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.433+0000 7f429428b700 1 -- 172.21.15.173:0/1526329079 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f428c060280 msgr2=0x7f428c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:00.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.433+0000 7f429428b700 1 --2- 172.21.15.173:0/1526329079 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f428c060280 0x7f428c0f90f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f4280009fb0 tx=0x7f4280009b30).stop 2024-01-21T19:55:00.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.433+0000 7f429428b700 1 -- 172.21.15.173:0/1526329079 shutdown_connections 2024-01-21T19:55:00.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.433+0000 7f429428b700 1 --2- 172.21.15.173:0/1526329079 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f428c060280 0x7f428c0f90f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:00.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.433+0000 7f429428b700 1 -- 172.21.15.173:0/1526329079 >> 172.21.15.173:0/1526329079 conn(0x7f428c0f0b70 msgr2=0x7f428c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:00.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.433+0000 7f429428b700 1 -- 172.21.15.173:0/1526329079 shutdown_connections 2024-01-21T19:55:00.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.433+0000 7f429428b700 1 -- 172.21.15.173:0/1526329079 wait complete. 2024-01-21T19:55:00.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.434+0000 7f429428b700 1 Processor -- start 2024-01-21T19:55:00.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.434+0000 7f429428b700 1 -- start start 2024-01-21T19:55:00.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.434+0000 7f429428b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f428c060280 0x7f428c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:00.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.434+0000 7f429428b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c0f4ff0 con 0x7f428c060280 2024-01-21T19:55:00.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.434+0000 7f4292027700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f428c060280 0x7f428c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:00.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.434+0000 7f4292027700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f428c060280 0x7f428c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:39994/0 (socket says 172.21.15.173:39994) 2024-01-21T19:55:00.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.434+0000 7f4292027700 1 -- 172.21.15.173:0/2104128942 learned_addr learned my addr 172.21.15.173:0/2104128942 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:00.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.434+0000 7f4292027700 1 -- 172.21.15.173:0/2104128942 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4280005040 con 0x7f428c060280 2024-01-21T19:55:00.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.434+0000 7f4292027700 1 --2- 172.21.15.173:0/2104128942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f428c060280 0x7f428c0f6b70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f4280009830 tx=0x7f428000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:00.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.435+0000 7f427effd700 1 -- 172.21.15.173:0/2104128942 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4280009320 con 0x7f428c060280 2024-01-21T19:55:00.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.435+0000 7f429428b700 1 -- 172.21.15.173:0/2104128942 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f428c0f5190 con 0x7f428c060280 2024-01-21T19:55:00.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.435+0000 7f427effd700 1 -- 172.21.15.173:0/2104128942 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42800136c0 con 0x7f428c060280 2024-01-21T19:55:00.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.435+0000 7f427effd700 1 -- 172.21.15.173:0/2104128942 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f428001ee90 con 0x7f428c060280 2024-01-21T19:55:00.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.435+0000 7f429428b700 1 -- 172.21.15.173:0/2104128942 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f428c0f5620 con 0x7f428c060280 2024-01-21T19:55:00.429 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.435+0000 7f427effd700 1 -- 172.21.15.173:0/2104128942 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f428001e420 con 0x7f428c060280 2024-01-21T19:55:00.429 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.435+0000 7f429428b700 1 -- 172.21.15.173:0/2104128942 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f428c059100 con 0x7f428c060280 2024-01-21T19:55:00.429 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.436+0000 7f427effd700 1 --2- 172.21.15.173:0/2104128942 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f427802ff10 0x7f42780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:00.429 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.436+0000 7f427effd700 1 -- 172.21.15.173:0/2104128942 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4280040f40 con 0x7f428c060280 2024-01-21T19:55:00.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.436+0000 7f4291826700 1 --2- 172.21.15.173:0/2104128942 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f427802ff10 0x7f42780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:00.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.436+0000 7f4291826700 1 --2- 172.21.15.173:0/2104128942 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f427802ff10 0x7f42780323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f42880096a0 tx=0x7f4288006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:00.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.440+0000 7f427effd700 1 -- 172.21.15.173:0/2104128942 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f428006d2b0 con 0x7f428c060280 2024-01-21T19:55:00.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.648+0000 7f429428b700 1 -- 172.21.15.173:0/2104128942 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f428c0fbf50 con 0x7f428c060280 2024-01-21T19:55:00.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.649+0000 7f427effd700 1 -- 172.21.15.173:0/2104128942 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f42800095f0 con 0x7f428c060280 2024-01-21T19:55:00.641 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:00.641 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:00.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.651+0000 7f429428b700 1 -- 172.21.15.173:0/2104128942 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f427802ff10 msgr2=0x7f42780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:00.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.651+0000 7f429428b700 1 --2- 172.21.15.173:0/2104128942 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f427802ff10 0x7f42780323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f42880096a0 tx=0x7f4288006b40).stop 2024-01-21T19:55:00.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.651+0000 7f429428b700 1 -- 172.21.15.173:0/2104128942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f428c060280 msgr2=0x7f428c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:00.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.651+0000 7f429428b700 1 --2- 172.21.15.173:0/2104128942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f428c060280 0x7f428c0f6b70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f4280009830 tx=0x7f428000b040).stop 2024-01-21T19:55:00.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.651+0000 7f429428b700 1 -- 172.21.15.173:0/2104128942 shutdown_connections 2024-01-21T19:55:00.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.651+0000 7f429428b700 1 --2- 172.21.15.173:0/2104128942 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f427802ff10 0x7f42780323d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:00.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.651+0000 7f429428b700 1 --2- 172.21.15.173:0/2104128942 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f428c060280 0x7f428c0f6b70 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:00.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.651+0000 7f429428b700 1 -- 172.21.15.173:0/2104128942 >> 172.21.15.173:0/2104128942 conn(0x7f428c0f0b70 msgr2=0x7f428c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:00.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.651+0000 7f429428b700 1 -- 172.21.15.173:0/2104128942 shutdown_connections 2024-01-21T19:55:00.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:00.651+0000 7f429428b700 1 -- 172.21.15.173:0/2104128942 wait complete. 2024-01-21T19:55:00.643 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:55:01.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:00 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:54:58.994226+0000 mgr.smithi169.espgue (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:01.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:00 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:00.649703+0000 mon.smithi169 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.173:0/2104128942' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:55:02.042 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:55:02.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:03.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:02 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:00.994582+0000 mgr.smithi169.espgue (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:03.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.228+0000 7f2ef46aa700 1 -- 172.21.15.173:0/1049073295 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2eec0f5410 msgr2=0x7f2eec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:03.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.228+0000 7f2ef46aa700 1 --2- 172.21.15.173:0/1049073295 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2eec0f5410 0x7f2eec0f5830 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f2edc009fb0 tx=0x7f2edc009b30).stop 2024-01-21T19:55:03.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.228+0000 7f2ef46aa700 1 -- 172.21.15.173:0/1049073295 shutdown_connections 2024-01-21T19:55:03.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.228+0000 7f2ef46aa700 1 --2- 172.21.15.173:0/1049073295 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2eec0f5410 0x7f2eec0f5830 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:03.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.228+0000 7f2ef46aa700 1 -- 172.21.15.173:0/1049073295 >> 172.21.15.173:0/1049073295 conn(0x7f2eec0f0b70 msgr2=0x7f2eec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:03.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.228+0000 7f2ef46aa700 1 -- 172.21.15.173:0/1049073295 shutdown_connections 2024-01-21T19:55:03.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.228+0000 7f2ef46aa700 1 -- 172.21.15.173:0/1049073295 wait complete. 2024-01-21T19:55:03.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.229+0000 7f2ef46aa700 1 Processor -- start 2024-01-21T19:55:03.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.229+0000 7f2ef46aa700 1 -- start start 2024-01-21T19:55:03.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.229+0000 7f2ef46aa700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2eec0f5410 0x7f2eec106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:03.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.229+0000 7f2ef46aa700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec106c00 con 0x7f2eec0f5410 2024-01-21T19:55:03.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.229+0000 7f2ef2446700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2eec0f5410 0x7f2eec106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:03.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.229+0000 7f2ef2446700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2eec0f5410 0x7f2eec106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:40012/0 (socket says 172.21.15.173:40012) 2024-01-21T19:55:03.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.229+0000 7f2ef2446700 1 -- 172.21.15.173:0/3213077134 learned_addr learned my addr 172.21.15.173:0/3213077134 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:03.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.229+0000 7f2ef2446700 1 -- 172.21.15.173:0/3213077134 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc005040 con 0x7f2eec0f5410 2024-01-21T19:55:03.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.229+0000 7f2ef2446700 1 --2- 172.21.15.173:0/3213077134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2eec0f5410 0x7f2eec106710 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f2edc013510 tx=0x7f2edc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:03.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.230+0000 7f2ee2ffd700 1 -- 172.21.15.173:0/3213077134 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2edc009320 con 0x7f2eec0f5410 2024-01-21T19:55:03.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.230+0000 7f2ef46aa700 1 -- 172.21.15.173:0/3213077134 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eec106e00 con 0x7f2eec0f5410 2024-01-21T19:55:03.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.230+0000 7f2ee2ffd700 1 -- 172.21.15.173:0/3213077134 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2edc013f00 con 0x7f2eec0f5410 2024-01-21T19:55:03.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.230+0000 7f2ee2ffd700 1 -- 172.21.15.173:0/3213077134 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2edc014d90 con 0x7f2eec0f5410 2024-01-21T19:55:03.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.230+0000 7f2ef46aa700 1 -- 172.21.15.173:0/3213077134 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eec107290 con 0x7f2eec0f5410 2024-01-21T19:55:03.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.230+0000 7f2ee2ffd700 1 -- 172.21.15.173:0/3213077134 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2edc014420 con 0x7f2eec0f5410 2024-01-21T19:55:03.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.230+0000 7f2ef46aa700 1 -- 172.21.15.173:0/3213077134 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eec045ad0 con 0x7f2eec0f5410 2024-01-21T19:55:03.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.231+0000 7f2ee2ffd700 1 --2- 172.21.15.173:0/3213077134 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2ed802ff60 0x7f2ed8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:03.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.231+0000 7f2ee2ffd700 1 -- 172.21.15.173:0/3213077134 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2edc041070 con 0x7f2eec0f5410 2024-01-21T19:55:03.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.231+0000 7f2ef1c45700 1 --2- 172.21.15.173:0/3213077134 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2ed802ff60 0x7f2ed8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:03.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.231+0000 7f2ef1c45700 1 --2- 172.21.15.173:0/3213077134 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2ed802ff60 0x7f2ed8032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f2ee80096a0 tx=0x7f2ee8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:03.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.235+0000 7f2ee2ffd700 1 -- 172.21.15.173:0/3213077134 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2edc0095f0 con 0x7f2eec0f5410 2024-01-21T19:55:03.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.440+0000 7f2ef46aa700 1 -- 172.21.15.173:0/3213077134 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2eec108140 con 0x7f2eec0f5410 2024-01-21T19:55:03.433 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.440+0000 7f2ee2ffd700 1 -- 172.21.15.173:0/3213077134 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2edc01db70 con 0x7f2eec0f5410 2024-01-21T19:55:03.433 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:03.433 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:03.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.442+0000 7f2ef46aa700 1 -- 172.21.15.173:0/3213077134 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2ed802ff60 msgr2=0x7f2ed8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:03.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.442+0000 7f2ef46aa700 1 --2- 172.21.15.173:0/3213077134 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2ed802ff60 0x7f2ed8032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f2ee80096a0 tx=0x7f2ee8006b40).stop 2024-01-21T19:55:03.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.442+0000 7f2ef46aa700 1 -- 172.21.15.173:0/3213077134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2eec0f5410 msgr2=0x7f2eec106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:03.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.442+0000 7f2ef46aa700 1 --2- 172.21.15.173:0/3213077134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2eec0f5410 0x7f2eec106710 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f2edc013510 tx=0x7f2edc00b040).stop 2024-01-21T19:55:03.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.442+0000 7f2ef46aa700 1 -- 172.21.15.173:0/3213077134 shutdown_connections 2024-01-21T19:55:03.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.442+0000 7f2ef46aa700 1 --2- 172.21.15.173:0/3213077134 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2ed802ff60 0x7f2ed8032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:03.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.442+0000 7f2ef46aa700 1 --2- 172.21.15.173:0/3213077134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2eec0f5410 0x7f2eec106710 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:03.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.442+0000 7f2ef46aa700 1 -- 172.21.15.173:0/3213077134 >> 172.21.15.173:0/3213077134 conn(0x7f2eec0f0b70 msgr2=0x7f2eec05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:03.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.442+0000 7f2ef46aa700 1 -- 172.21.15.173:0/3213077134 shutdown_connections 2024-01-21T19:55:03.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:03.442+0000 7f2ef46aa700 1 -- 172.21.15.173:0/3213077134 wait complete. 2024-01-21T19:55:03.436 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:55:04.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:03 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:03.442156+0000 mon.smithi169 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.173:0/3213077134' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:55:04.835 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:55:04.835 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:05.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:04 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:02.994936+0000 mgr.smithi169.espgue (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:06.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.046+0000 7f038f30e700 1 -- 172.21.15.173:0/3568161886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f03880f5410 msgr2=0x7f03880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:06.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.046+0000 7f038f30e700 1 --2- 172.21.15.173:0/3568161886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f03880f5410 0x7f03880f5830 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f0378009fb0 tx=0x7f0378009b30).stop 2024-01-21T19:55:06.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.046+0000 7f038f30e700 1 -- 172.21.15.173:0/3568161886 shutdown_connections 2024-01-21T19:55:06.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.046+0000 7f038f30e700 1 --2- 172.21.15.173:0/3568161886 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f03880f5410 0x7f03880f5830 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:06.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.046+0000 7f038f30e700 1 -- 172.21.15.173:0/3568161886 >> 172.21.15.173:0/3568161886 conn(0x7f03880f0b70 msgr2=0x7f03880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:06.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.046+0000 7f038f30e700 1 -- 172.21.15.173:0/3568161886 shutdown_connections 2024-01-21T19:55:06.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.046+0000 7f038f30e700 1 -- 172.21.15.173:0/3568161886 wait complete. 2024-01-21T19:55:06.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.047+0000 7f038f30e700 1 Processor -- start 2024-01-21T19:55:06.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.047+0000 7f038f30e700 1 -- start start 2024-01-21T19:55:06.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.047+0000 7f038f30e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f03880f5410 0x7f0388105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:06.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.047+0000 7f038f30e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0388105b20 con 0x7f03880f5410 2024-01-21T19:55:06.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.047+0000 7f038d0aa700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f03880f5410 0x7f0388105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:06.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.047+0000 7f038d0aa700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f03880f5410 0x7f0388105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:41480/0 (socket says 172.21.15.173:41480) 2024-01-21T19:55:06.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.047+0000 7f038d0aa700 1 -- 172.21.15.173:0/4158801614 learned_addr learned my addr 172.21.15.173:0/4158801614 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:06.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.048+0000 7f038d0aa700 1 -- 172.21.15.173:0/4158801614 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0378005040 con 0x7f03880f5410 2024-01-21T19:55:06.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.048+0000 7f038d0aa700 1 --2- 172.21.15.173:0/4158801614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f03880f5410 0x7f0388105630 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f0378013510 tx=0x7f037800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:06.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.048+0000 7f037dffb700 1 -- 172.21.15.173:0/4158801614 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0378009320 con 0x7f03880f5410 2024-01-21T19:55:06.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.048+0000 7f037dffb700 1 -- 172.21.15.173:0/4158801614 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0378013f00 con 0x7f03880f5410 2024-01-21T19:55:06.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.048+0000 7f038f30e700 1 -- 172.21.15.173:0/4158801614 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0388105d20 con 0x7f03880f5410 2024-01-21T19:55:06.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.048+0000 7f037dffb700 1 -- 172.21.15.173:0/4158801614 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0378014d90 con 0x7f03880f5410 2024-01-21T19:55:06.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.048+0000 7f038f30e700 1 -- 172.21.15.173:0/4158801614 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03881061b0 con 0x7f03880f5410 2024-01-21T19:55:06.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.049+0000 7f037dffb700 1 -- 172.21.15.173:0/4158801614 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f03780263a0 con 0x7f03880f5410 2024-01-21T19:55:06.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.049+0000 7f038f30e700 1 -- 172.21.15.173:0/4158801614 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0388045ad0 con 0x7f03880f5410 2024-01-21T19:55:06.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.049+0000 7f037dffb700 1 --2- 172.21.15.173:0/4158801614 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f037402ff60 0x7f0374032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:06.046 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.049+0000 7f037dffb700 1 -- 172.21.15.173:0/4158801614 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0378041370 con 0x7f03880f5410 2024-01-21T19:55:06.046 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.050+0000 7f038c8a9700 1 --2- 172.21.15.173:0/4158801614 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f037402ff60 0x7f0374032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:06.046 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.050+0000 7f038c8a9700 1 --2- 172.21.15.173:0/4158801614 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f037402ff60 0x7f0374032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f03840096a0 tx=0x7f0384006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:06.047 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.054+0000 7f037dffb700 1 -- 172.21.15.173:0/4158801614 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f03780126f0 con 0x7f03880f5410 2024-01-21T19:55:06.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.258+0000 7f038f30e700 1 -- 172.21.15.173:0/4158801614 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f03880fae60 con 0x7f03880f5410 2024-01-21T19:55:06.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.259+0000 7f037dffb700 1 -- 172.21.15.173:0/4158801614 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0378012330 con 0x7f03880f5410 2024-01-21T19:55:06.252 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:06.252 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:06.253 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.260+0000 7f038f30e700 1 -- 172.21.15.173:0/4158801614 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f037402ff60 msgr2=0x7f0374032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:06.253 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.260+0000 7f038f30e700 1 --2- 172.21.15.173:0/4158801614 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f037402ff60 0x7f0374032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f03840096a0 tx=0x7f0384006b40).stop 2024-01-21T19:55:06.254 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.260+0000 7f038f30e700 1 -- 172.21.15.173:0/4158801614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f03880f5410 msgr2=0x7f0388105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:06.254 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.260+0000 7f038f30e700 1 --2- 172.21.15.173:0/4158801614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f03880f5410 0x7f0388105630 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f0378013510 tx=0x7f037800b040).stop 2024-01-21T19:55:06.254 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.260+0000 7f038f30e700 1 -- 172.21.15.173:0/4158801614 shutdown_connections 2024-01-21T19:55:06.254 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.260+0000 7f038f30e700 1 --2- 172.21.15.173:0/4158801614 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f037402ff60 0x7f0374032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:06.254 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.261+0000 7f038f30e700 1 --2- 172.21.15.173:0/4158801614 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f03880f5410 0x7f0388105630 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:06.254 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.261+0000 7f038f30e700 1 -- 172.21.15.173:0/4158801614 >> 172.21.15.173:0/4158801614 conn(0x7f03880f0b70 msgr2=0x7f03880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:06.254 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.261+0000 7f038f30e700 1 -- 172.21.15.173:0/4158801614 shutdown_connections 2024-01-21T19:55:06.254 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:06.261+0000 7f038f30e700 1 -- 172.21.15.173:0/4158801614 wait complete. 2024-01-21T19:55:06.254 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:55:07.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:06 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:04.995280+0000 mgr.smithi169.espgue (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:07.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:06 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:06.260391+0000 mon.smithi169 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.173:0/4158801614' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:55:07.653 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:55:07.653 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:08.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.905+0000 7f0ce056e700 1 -- 172.21.15.173:0/4108324659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cd8060280 msgr2=0x7f0cd80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:08.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.905+0000 7f0ce056e700 1 --2- 172.21.15.173:0/4108324659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cd8060280 0x7f0cd80f90f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f0cd4009fb0 tx=0x7f0cd4009b30).stop 2024-01-21T19:55:08.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.905+0000 7f0ce056e700 1 -- 172.21.15.173:0/4108324659 shutdown_connections 2024-01-21T19:55:08.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.905+0000 7f0ce056e700 1 --2- 172.21.15.173:0/4108324659 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cd8060280 0x7f0cd80f90f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:08.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.905+0000 7f0ce056e700 1 -- 172.21.15.173:0/4108324659 >> 172.21.15.173:0/4108324659 conn(0x7f0cd80f0b70 msgr2=0x7f0cd80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:08.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.905+0000 7f0ce056e700 1 -- 172.21.15.173:0/4108324659 shutdown_connections 2024-01-21T19:55:08.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.905+0000 7f0ce056e700 1 -- 172.21.15.173:0/4108324659 wait complete. 2024-01-21T19:55:08.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.906+0000 7f0ce056e700 1 Processor -- start 2024-01-21T19:55:08.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.906+0000 7f0ce056e700 1 -- start start 2024-01-21T19:55:08.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.906+0000 7f0ce056e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cd8060280 0x7f0cd80f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:08.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.906+0000 7f0ce056e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd80f4ff0 con 0x7f0cd8060280 2024-01-21T19:55:08.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.906+0000 7f0cde30a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cd8060280 0x7f0cd80f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:08.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.906+0000 7f0cde30a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cd8060280 0x7f0cd80f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:41494/0 (socket says 172.21.15.173:41494) 2024-01-21T19:55:08.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.906+0000 7f0cde30a700 1 -- 172.21.15.173:0/2659339588 learned_addr learned my addr 172.21.15.173:0/2659339588 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:08.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.906+0000 7f0cde30a700 1 -- 172.21.15.173:0/2659339588 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd4005040 con 0x7f0cd8060280 2024-01-21T19:55:08.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.906+0000 7f0cde30a700 1 --2- 172.21.15.173:0/2659339588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cd8060280 0x7f0cd80f6b70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f0cd4013510 tx=0x7f0cd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:08.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.907+0000 7f0cceffd700 1 -- 172.21.15.173:0/2659339588 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cd4009320 con 0x7f0cd8060280 2024-01-21T19:55:08.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.907+0000 7f0cceffd700 1 -- 172.21.15.173:0/2659339588 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0cd4013f00 con 0x7f0cd8060280 2024-01-21T19:55:08.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.907+0000 7f0ce056e700 1 -- 172.21.15.173:0/2659339588 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd80f5190 con 0x7f0cd8060280 2024-01-21T19:55:08.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.907+0000 7f0cceffd700 1 -- 172.21.15.173:0/2659339588 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cd4014dc0 con 0x7f0cd8060280 2024-01-21T19:55:08.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.907+0000 7f0ce056e700 1 -- 172.21.15.173:0/2659339588 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd80f5620 con 0x7f0cd8060280 2024-01-21T19:55:08.902 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.907+0000 7f0cceffd700 1 -- 172.21.15.173:0/2659339588 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0cd40095f0 con 0x7f0cd8060280 2024-01-21T19:55:08.902 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.907+0000 7f0ce056e700 1 -- 172.21.15.173:0/2659339588 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd8059100 con 0x7f0cd8060280 2024-01-21T19:55:08.902 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.908+0000 7f0cceffd700 1 --2- 172.21.15.173:0/2659339588 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0cc402ffb0 0x7f0cc4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:08.902 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.908+0000 7f0cceffd700 1 -- 172.21.15.173:0/2659339588 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0cd40410c0 con 0x7f0cd8060280 2024-01-21T19:55:08.902 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.908+0000 7f0cddb09700 1 --2- 172.21.15.173:0/2659339588 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0cc402ffb0 0x7f0cc4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:08.902 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.908+0000 7f0cddb09700 1 --2- 172.21.15.173:0/2659339588 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0cc402ffb0 0x7f0cc4032470 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f0cc80096a0 tx=0x7f0cc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:08.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:08.912+0000 7f0cceffd700 1 -- 172.21.15.173:0/2659339588 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0cd401c790 con 0x7f0cd8060280 2024-01-21T19:55:09.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:09.116+0000 7f0ce056e700 1 -- 172.21.15.173:0/2659339588 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0cd80fbf50 con 0x7f0cd8060280 2024-01-21T19:55:09.109 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:09.116+0000 7f0cceffd700 1 -- 172.21.15.173:0/2659339588 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0cd4010230 con 0x7f0cd8060280 2024-01-21T19:55:09.109 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:09.109 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:09.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:09.118+0000 7f0ce056e700 1 -- 172.21.15.173:0/2659339588 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0cc402ffb0 msgr2=0x7f0cc4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:09.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:09.118+0000 7f0ce056e700 1 --2- 172.21.15.173:0/2659339588 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0cc402ffb0 0x7f0cc4032470 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f0cc80096a0 tx=0x7f0cc8006b40).stop 2024-01-21T19:55:09.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:09.118+0000 7f0ce056e700 1 -- 172.21.15.173:0/2659339588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cd8060280 msgr2=0x7f0cd80f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:09.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:09.118+0000 7f0ce056e700 1 --2- 172.21.15.173:0/2659339588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cd8060280 0x7f0cd80f6b70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f0cd4013510 tx=0x7f0cd400b040).stop 2024-01-21T19:55:09.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:09.118+0000 7f0ce056e700 1 -- 172.21.15.173:0/2659339588 shutdown_connections 2024-01-21T19:55:09.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:09.118+0000 7f0ce056e700 1 --2- 172.21.15.173:0/2659339588 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0cc402ffb0 0x7f0cc4032470 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:09.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:09.118+0000 7f0ce056e700 1 --2- 172.21.15.173:0/2659339588 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0cd8060280 0x7f0cd80f6b70 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:09.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:09.118+0000 7f0ce056e700 1 -- 172.21.15.173:0/2659339588 >> 172.21.15.173:0/2659339588 conn(0x7f0cd80f0b70 msgr2=0x7f0cd80f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:09.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:09.118+0000 7f0ce056e700 1 -- 172.21.15.173:0/2659339588 shutdown_connections 2024-01-21T19:55:09.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:09.118+0000 7f0ce056e700 1 -- 172.21.15.173:0/2659339588 wait complete. 2024-01-21T19:55:09.111 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:55:09.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:08 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:06.995628+0000 mgr.smithi169.espgue (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:10.021 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:09 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:09.118179+0000 mon.smithi169 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.173:0/2659339588' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:55:10.506 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:55:10.507 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:11.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:10 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:08.996027+0000 mgr.smithi169.espgue (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:11.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.698+0000 7f64b334f700 1 -- 172.21.15.173:0/518913793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64ac060280 msgr2=0x7f64ac0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:11.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.698+0000 7f64b334f700 1 --2- 172.21.15.173:0/518913793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64ac060280 0x7f64ac0f90f0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f649c009fb0 tx=0x7f649c009b30).stop 2024-01-21T19:55:11.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.698+0000 7f64b334f700 1 -- 172.21.15.173:0/518913793 shutdown_connections 2024-01-21T19:55:11.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.698+0000 7f64b334f700 1 --2- 172.21.15.173:0/518913793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64ac060280 0x7f64ac0f90f0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:11.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.698+0000 7f64b334f700 1 -- 172.21.15.173:0/518913793 >> 172.21.15.173:0/518913793 conn(0x7f64ac0f0b70 msgr2=0x7f64ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:11.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.698+0000 7f64b334f700 1 -- 172.21.15.173:0/518913793 shutdown_connections 2024-01-21T19:55:11.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.698+0000 7f64b334f700 1 -- 172.21.15.173:0/518913793 wait complete. 2024-01-21T19:55:11.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.699+0000 7f64b334f700 1 Processor -- start 2024-01-21T19:55:11.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.699+0000 7f64b334f700 1 -- start start 2024-01-21T19:55:11.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.699+0000 7f64b334f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64ac060280 0x7f64ac0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:11.694 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.699+0000 7f64b334f700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac0f4ff0 con 0x7f64ac060280 2024-01-21T19:55:11.694 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.699+0000 7f64b10eb700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64ac060280 0x7f64ac0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:11.694 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.699+0000 7f64b10eb700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64ac060280 0x7f64ac0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:41514/0 (socket says 172.21.15.173:41514) 2024-01-21T19:55:11.694 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.699+0000 7f64b10eb700 1 -- 172.21.15.173:0/55203543 learned_addr learned my addr 172.21.15.173:0/55203543 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:11.694 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.700+0000 7f64b10eb700 1 -- 172.21.15.173:0/55203543 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649c005040 con 0x7f64ac060280 2024-01-21T19:55:11.694 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.700+0000 7f64b10eb700 1 --2- 172.21.15.173:0/55203543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64ac060280 0x7f64ac0f6b70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f649c009830 tx=0x7f649c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:11.694 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.700+0000 7f64a1ffb700 1 -- 172.21.15.173:0/55203543 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f649c009320 con 0x7f64ac060280 2024-01-21T19:55:11.694 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.700+0000 7f64b334f700 1 -- 172.21.15.173:0/55203543 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64ac0f5190 con 0x7f64ac060280 2024-01-21T19:55:11.694 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.700+0000 7f64a1ffb700 1 -- 172.21.15.173:0/55203543 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f649c0136c0 con 0x7f64ac060280 2024-01-21T19:55:11.694 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.700+0000 7f64a1ffb700 1 -- 172.21.15.173:0/55203543 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f649c01ee90 con 0x7f64ac060280 2024-01-21T19:55:11.694 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.700+0000 7f64b334f700 1 -- 172.21.15.173:0/55203543 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64ac0f5620 con 0x7f64ac060280 2024-01-21T19:55:11.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.701+0000 7f64b334f700 1 -- 172.21.15.173:0/55203543 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64ac059100 con 0x7f64ac060280 2024-01-21T19:55:11.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.701+0000 7f64a1ffb700 1 -- 172.21.15.173:0/55203543 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f649c01e420 con 0x7f64ac060280 2024-01-21T19:55:11.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.701+0000 7f64a1ffb700 1 --2- 172.21.15.173:0/55203543 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f649802ff10 0x7f64980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:11.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.701+0000 7f64a1ffb700 1 -- 172.21.15.173:0/55203543 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f649c040fa0 con 0x7f64ac060280 2024-01-21T19:55:11.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.701+0000 7f64b08ea700 1 --2- 172.21.15.173:0/55203543 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f649802ff10 0x7f64980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:11.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.702+0000 7f64b08ea700 1 --2- 172.21.15.173:0/55203543 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f649802ff10 0x7f64980323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f64a80096a0 tx=0x7f64a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:11.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.706+0000 7f64a1ffb700 1 -- 172.21.15.173:0/55203543 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f649c0095f0 con 0x7f64ac060280 2024-01-21T19:55:11.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.912+0000 7f64b334f700 1 -- 172.21.15.173:0/55203543 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64ac045ad0 con 0x7f64ac060280 2024-01-21T19:55:11.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.913+0000 7f64a1ffb700 1 -- 172.21.15.173:0/55203543 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f649c02c2d0 con 0x7f64ac060280 2024-01-21T19:55:11.905 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:11.905 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:11.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.914+0000 7f64b334f700 1 -- 172.21.15.173:0/55203543 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f649802ff10 msgr2=0x7f64980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:11.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.914+0000 7f64b334f700 1 --2- 172.21.15.173:0/55203543 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f649802ff10 0x7f64980323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f64a80096a0 tx=0x7f64a8006b40).stop 2024-01-21T19:55:11.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.914+0000 7f64b334f700 1 -- 172.21.15.173:0/55203543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64ac060280 msgr2=0x7f64ac0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:11.908 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.914+0000 7f64b334f700 1 --2- 172.21.15.173:0/55203543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64ac060280 0x7f64ac0f6b70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f649c009830 tx=0x7f649c00b040).stop 2024-01-21T19:55:11.908 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.914+0000 7f64b334f700 1 -- 172.21.15.173:0/55203543 shutdown_connections 2024-01-21T19:55:11.908 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.914+0000 7f64b334f700 1 --2- 172.21.15.173:0/55203543 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f649802ff10 0x7f64980323d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:11.908 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.914+0000 7f64b334f700 1 --2- 172.21.15.173:0/55203543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f64ac060280 0x7f64ac0f6b70 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:11.908 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.914+0000 7f64b334f700 1 -- 172.21.15.173:0/55203543 >> 172.21.15.173:0/55203543 conn(0x7f64ac0f0b70 msgr2=0x7f64ac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:11.908 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.914+0000 7f64b334f700 1 -- 172.21.15.173:0/55203543 shutdown_connections 2024-01-21T19:55:11.908 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:11.914+0000 7f64b334f700 1 -- 172.21.15.173:0/55203543 wait complete. 2024-01-21T19:55:11.908 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:55:13.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:12 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:10.996362+0000 mgr.smithi169.espgue (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:13.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:12 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:11.914388+0000 mon.smithi169 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.173:0/55203543' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:55:13.316 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:55:13.317 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:14.531 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.538+0000 7f4de8096700 1 -- 172.21.15.173:0/3697404860 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4de00f5410 msgr2=0x7f4de00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:14.531 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.538+0000 7f4de8096700 1 --2- 172.21.15.173:0/3697404860 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4de00f5410 0x7f4de00f5830 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f4dd8009960 tx=0x7f4dd8004d10).stop 2024-01-21T19:55:14.531 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.538+0000 7f4de8096700 1 -- 172.21.15.173:0/3697404860 shutdown_connections 2024-01-21T19:55:14.531 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.538+0000 7f4de8096700 1 --2- 172.21.15.173:0/3697404860 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4de00f5410 0x7f4de00f5830 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:14.531 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.538+0000 7f4de8096700 1 -- 172.21.15.173:0/3697404860 >> 172.21.15.173:0/3697404860 conn(0x7f4de00f0b70 msgr2=0x7f4de00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:14.531 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.538+0000 7f4de8096700 1 -- 172.21.15.173:0/3697404860 shutdown_connections 2024-01-21T19:55:14.531 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.538+0000 7f4de8096700 1 -- 172.21.15.173:0/3697404860 wait complete. 2024-01-21T19:55:14.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.539+0000 7f4de8096700 1 Processor -- start 2024-01-21T19:55:14.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.539+0000 7f4de8096700 1 -- start start 2024-01-21T19:55:14.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.539+0000 7f4de8096700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4de00f5410 0x7f4de0105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:14.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.539+0000 7f4de8096700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de0105af0 con 0x7f4de00f5410 2024-01-21T19:55:14.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.540+0000 7f4de5e32700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4de00f5410 0x7f4de0105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:14.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.540+0000 7f4de5e32700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4de00f5410 0x7f4de0105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:43848/0 (socket says 172.21.15.173:43848) 2024-01-21T19:55:14.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.540+0000 7f4de5e32700 1 -- 172.21.15.173:0/3132143260 learned_addr learned my addr 172.21.15.173:0/3132143260 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:14.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.540+0000 7f4de5e32700 1 -- 172.21.15.173:0/3132143260 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd8009640 con 0x7f4de00f5410 2024-01-21T19:55:14.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.540+0000 7f4de5e32700 1 --2- 172.21.15.173:0/3132143260 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4de00f5410 0x7f4de0105600 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f4dd8004b70 tx=0x7f4dd8004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:14.534 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.540+0000 7f4dd67fc700 1 -- 172.21.15.173:0/3132143260 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4dd8018070 con 0x7f4de00f5410 2024-01-21T19:55:14.534 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.540+0000 7f4de8096700 1 -- 172.21.15.173:0/3132143260 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de0105cf0 con 0x7f4de00f5410 2024-01-21T19:55:14.534 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.540+0000 7f4dd67fc700 1 -- 172.21.15.173:0/3132143260 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4dd800b870 con 0x7f4de00f5410 2024-01-21T19:55:14.534 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.540+0000 7f4dd67fc700 1 -- 172.21.15.173:0/3132143260 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4dd801c560 con 0x7f4de00f5410 2024-01-21T19:55:14.534 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.540+0000 7f4de8096700 1 -- 172.21.15.173:0/3132143260 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de0106180 con 0x7f4de00f5410 2024-01-21T19:55:14.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.541+0000 7f4dd67fc700 1 -- 172.21.15.173:0/3132143260 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4dd801c6c0 con 0x7f4de00f5410 2024-01-21T19:55:14.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.541+0000 7f4de8096700 1 -- 172.21.15.173:0/3132143260 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de0045ad0 con 0x7f4de00f5410 2024-01-21T19:55:14.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.541+0000 7f4dd67fc700 1 --2- 172.21.15.173:0/3132143260 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4dcc02ffb0 0x7f4dcc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:14.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.541+0000 7f4dd67fc700 1 -- 172.21.15.173:0/3132143260 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4dd80414d0 con 0x7f4de00f5410 2024-01-21T19:55:14.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.541+0000 7f4de5631700 1 --2- 172.21.15.173:0/3132143260 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4dcc02ffb0 0x7f4dcc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:14.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.542+0000 7f4de5631700 1 --2- 172.21.15.173:0/3132143260 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4dcc02ffb0 0x7f4dcc032470 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f4dd00096a0 tx=0x7f4dd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:14.538 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.546+0000 7f4dd67fc700 1 -- 172.21.15.173:0/3132143260 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4dd8012560 con 0x7f4de00f5410 2024-01-21T19:55:14.742 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.749+0000 7f4de8096700 1 -- 172.21.15.173:0/3132143260 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4de00fae60 con 0x7f4de00f5410 2024-01-21T19:55:14.742 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.750+0000 7f4dd67fc700 1 -- 172.21.15.173:0/3132143260 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4dd8012380 con 0x7f4de00f5410 2024-01-21T19:55:14.742 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:14.742 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:14.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.751+0000 7f4de8096700 1 -- 172.21.15.173:0/3132143260 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4dcc02ffb0 msgr2=0x7f4dcc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:14.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.751+0000 7f4de8096700 1 --2- 172.21.15.173:0/3132143260 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4dcc02ffb0 0x7f4dcc032470 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f4dd00096a0 tx=0x7f4dd0006b40).stop 2024-01-21T19:55:14.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.751+0000 7f4de8096700 1 -- 172.21.15.173:0/3132143260 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4de00f5410 msgr2=0x7f4de0105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:14.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.751+0000 7f4de8096700 1 --2- 172.21.15.173:0/3132143260 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4de00f5410 0x7f4de0105600 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f4dd8004b70 tx=0x7f4dd8004d10).stop 2024-01-21T19:55:14.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.752+0000 7f4de8096700 1 -- 172.21.15.173:0/3132143260 shutdown_connections 2024-01-21T19:55:14.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.752+0000 7f4de8096700 1 --2- 172.21.15.173:0/3132143260 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4dcc02ffb0 0x7f4dcc032470 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:14.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.752+0000 7f4de8096700 1 --2- 172.21.15.173:0/3132143260 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4de00f5410 0x7f4de0105600 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:14.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.752+0000 7f4de8096700 1 -- 172.21.15.173:0/3132143260 >> 172.21.15.173:0/3132143260 conn(0x7f4de00f0b70 msgr2=0x7f4de00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:14.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.752+0000 7f4de8096700 1 -- 172.21.15.173:0/3132143260 shutdown_connections 2024-01-21T19:55:14.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:14.752+0000 7f4de8096700 1 -- 172.21.15.173:0/3132143260 wait complete. 2024-01-21T19:55:14.745 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:55:15.167 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:14 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:12.996713+0000 mgr.smithi169.espgue (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:15.167 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:14 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:14.751681+0000 mon.smithi169 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.173:0/3132143260' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:55:16.145 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:55:16.145 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:17.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:16 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:14.997034+0000 mgr.smithi169.espgue (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:17.337 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.343+0000 7f9d120e7700 1 -- 172.21.15.173:0/3611547757 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d0c0f6510 msgr2=0x7f9d0c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:17.337 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.343+0000 7f9d120e7700 1 --2- 172.21.15.173:0/3611547757 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d0c0f6510 0x7f9d0c0f6930 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f9cf4009fb0 tx=0x7f9cf4009b30).stop 2024-01-21T19:55:17.337 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.343+0000 7f9d120e7700 1 -- 172.21.15.173:0/3611547757 shutdown_connections 2024-01-21T19:55:17.337 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.343+0000 7f9d120e7700 1 --2- 172.21.15.173:0/3611547757 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d0c0f6510 0x7f9d0c0f6930 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:17.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.343+0000 7f9d120e7700 1 -- 172.21.15.173:0/3611547757 >> 172.21.15.173:0/3611547757 conn(0x7f9d0c0f1c70 msgr2=0x7f9d0c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:17.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.343+0000 7f9d120e7700 1 -- 172.21.15.173:0/3611547757 shutdown_connections 2024-01-21T19:55:17.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.343+0000 7f9d120e7700 1 -- 172.21.15.173:0/3611547757 wait complete. 2024-01-21T19:55:17.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.344+0000 7f9d120e7700 1 Processor -- start 2024-01-21T19:55:17.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.344+0000 7f9d120e7700 1 -- start start 2024-01-21T19:55:17.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.344+0000 7f9d120e7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d0c0f6510 0x7f9d0c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:17.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.344+0000 7f9d120e7700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c069e20 con 0x7f9d0c0f6510 2024-01-21T19:55:17.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.344+0000 7f9d0b7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d0c0f6510 0x7f9d0c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:17.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.344+0000 7f9d0b7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d0c0f6510 0x7f9d0c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:43866/0 (socket says 172.21.15.173:43866) 2024-01-21T19:55:17.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.344+0000 7f9d0b7fe700 1 -- 172.21.15.173:0/1038245433 learned_addr learned my addr 172.21.15.173:0/1038245433 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:17.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.344+0000 7f9d0b7fe700 1 -- 172.21.15.173:0/1038245433 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf4005040 con 0x7f9d0c0f6510 2024-01-21T19:55:17.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.345+0000 7f9d0b7fe700 1 --2- 172.21.15.173:0/1038245433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d0c0f6510 0x7f9d0c06b9a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f9cf4013510 tx=0x7f9cf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:17.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.345+0000 7f9d03fff700 1 -- 172.21.15.173:0/1038245433 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cf4009320 con 0x7f9d0c0f6510 2024-01-21T19:55:17.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.345+0000 7f9d120e7700 1 -- 172.21.15.173:0/1038245433 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0c069fc0 con 0x7f9d0c0f6510 2024-01-21T19:55:17.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.345+0000 7f9d03fff700 1 -- 172.21.15.173:0/1038245433 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9cf4013f00 con 0x7f9d0c0f6510 2024-01-21T19:55:17.339 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.345+0000 7f9d03fff700 1 -- 172.21.15.173:0/1038245433 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cf4014d90 con 0x7f9d0c0f6510 2024-01-21T19:55:17.339 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.345+0000 7f9d120e7700 1 -- 172.21.15.173:0/1038245433 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d0c06a450 con 0x7f9d0c0f6510 2024-01-21T19:55:17.339 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.346+0000 7f9d03fff700 1 -- 172.21.15.173:0/1038245433 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9cf4014420 con 0x7f9d0c0f6510 2024-01-21T19:55:17.339 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.346+0000 7f9d120e7700 1 -- 172.21.15.173:0/1038245433 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d0c059100 con 0x7f9d0c0f6510 2024-01-21T19:55:17.340 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.346+0000 7f9d03fff700 1 --2- 172.21.15.173:0/1038245433 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9cf802ff60 0x7f9cf8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:17.340 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.346+0000 7f9d03fff700 1 -- 172.21.15.173:0/1038245433 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9cf4041000 con 0x7f9d0c0f6510 2024-01-21T19:55:17.340 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.346+0000 7f9d0affd700 1 --2- 172.21.15.173:0/1038245433 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9cf802ff60 0x7f9cf8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:17.340 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.346+0000 7f9d0affd700 1 --2- 172.21.15.173:0/1038245433 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9cf802ff60 0x7f9cf8032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f9cfc0096a0 tx=0x7f9cfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:17.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.350+0000 7f9d03fff700 1 -- 172.21.15.173:0/1038245433 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9cf40095f0 con 0x7f9d0c0f6510 2024-01-21T19:55:17.549 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.557+0000 7f9d120e7700 1 -- 172.21.15.173:0/1038245433 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d0c0fbf60 con 0x7f9d0c0f6510 2024-01-21T19:55:17.550 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.558+0000 7f9d03fff700 1 -- 172.21.15.173:0/1038245433 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9cf401db70 con 0x7f9d0c0f6510 2024-01-21T19:55:17.550 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:17.550 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:17.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.559+0000 7f9d120e7700 1 -- 172.21.15.173:0/1038245433 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9cf802ff60 msgr2=0x7f9cf8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:17.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.559+0000 7f9d120e7700 1 --2- 172.21.15.173:0/1038245433 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9cf802ff60 0x7f9cf8032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f9cfc0096a0 tx=0x7f9cfc006b40).stop 2024-01-21T19:55:17.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.559+0000 7f9d120e7700 1 -- 172.21.15.173:0/1038245433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d0c0f6510 msgr2=0x7f9d0c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:17.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.559+0000 7f9d120e7700 1 --2- 172.21.15.173:0/1038245433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d0c0f6510 0x7f9d0c06b9a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f9cf4013510 tx=0x7f9cf400b040).stop 2024-01-21T19:55:17.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.559+0000 7f9d120e7700 1 -- 172.21.15.173:0/1038245433 shutdown_connections 2024-01-21T19:55:17.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.559+0000 7f9d120e7700 1 --2- 172.21.15.173:0/1038245433 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9cf802ff60 0x7f9cf8032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:17.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.559+0000 7f9d120e7700 1 --2- 172.21.15.173:0/1038245433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9d0c0f6510 0x7f9d0c06b9a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:17.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.559+0000 7f9d120e7700 1 -- 172.21.15.173:0/1038245433 >> 172.21.15.173:0/1038245433 conn(0x7f9d0c0f1c70 msgr2=0x7f9d0c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:17.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.559+0000 7f9d120e7700 1 -- 172.21.15.173:0/1038245433 shutdown_connections 2024-01-21T19:55:17.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:17.559+0000 7f9d120e7700 1 -- 172.21.15.173:0/1038245433 wait complete. 2024-01-21T19:55:17.553 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:55:18.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:17 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:17.559439+0000 mon.smithi169 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.173:0/1038245433' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:55:18.931 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:55:18.931 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:19.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:18 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:16.997331+0000 mgr.smithi169.espgue (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:20.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.147+0000 7f997f9ed700 1 -- 172.21.15.173:0/3176446103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99780f5410 msgr2=0x7f99780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:20.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.147+0000 7f997f9ed700 1 --2- 172.21.15.173:0/3176446103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99780f5410 0x7f99780f5830 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f9974009fb0 tx=0x7f9974009b30).stop 2024-01-21T19:55:20.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.147+0000 7f997f9ed700 1 -- 172.21.15.173:0/3176446103 shutdown_connections 2024-01-21T19:55:20.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.147+0000 7f997f9ed700 1 --2- 172.21.15.173:0/3176446103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99780f5410 0x7f99780f5830 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:20.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.147+0000 7f997f9ed700 1 -- 172.21.15.173:0/3176446103 >> 172.21.15.173:0/3176446103 conn(0x7f99780f0b70 msgr2=0x7f99780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:20.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.147+0000 7f997f9ed700 1 -- 172.21.15.173:0/3176446103 shutdown_connections 2024-01-21T19:55:20.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.147+0000 7f997f9ed700 1 -- 172.21.15.173:0/3176446103 wait complete. 2024-01-21T19:55:20.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.148+0000 7f997f9ed700 1 Processor -- start 2024-01-21T19:55:20.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.148+0000 7f997f9ed700 1 -- start start 2024-01-21T19:55:20.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.148+0000 7f997f9ed700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99780f5410 0x7f9978109390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:20.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.148+0000 7f997f9ed700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9978060280 con 0x7f99780f5410 2024-01-21T19:55:20.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.148+0000 7f997d789700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99780f5410 0x7f9978109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:20.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.148+0000 7f997d789700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99780f5410 0x7f9978109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:43890/0 (socket says 172.21.15.173:43890) 2024-01-21T19:55:20.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.148+0000 7f997d789700 1 -- 172.21.15.173:0/3776119932 learned_addr learned my addr 172.21.15.173:0/3776119932 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:20.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.148+0000 7f997d789700 1 -- 172.21.15.173:0/3776119932 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9974005040 con 0x7f99780f5410 2024-01-21T19:55:20.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.149+0000 7f997d789700 1 --2- 172.21.15.173:0/3776119932 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99780f5410 0x7f9978109390 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f9974013510 tx=0x7f997400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:20.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.149+0000 7f996dffb700 1 -- 172.21.15.173:0/3776119932 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9974009320 con 0x7f99780f5410 2024-01-21T19:55:20.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.149+0000 7f997f9ed700 1 -- 172.21.15.173:0/3776119932 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9978060480 con 0x7f99780f5410 2024-01-21T19:55:20.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.149+0000 7f996dffb700 1 -- 172.21.15.173:0/3776119932 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9974013f00 con 0x7f99780f5410 2024-01-21T19:55:20.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.149+0000 7f996dffb700 1 -- 172.21.15.173:0/3776119932 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9974014d90 con 0x7f99780f5410 2024-01-21T19:55:20.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.149+0000 7f997f9ed700 1 -- 172.21.15.173:0/3776119932 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9978109ac0 con 0x7f99780f5410 2024-01-21T19:55:20.143 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.150+0000 7f997f9ed700 1 -- 172.21.15.173:0/3776119932 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9978045ad0 con 0x7f99780f5410 2024-01-21T19:55:20.143 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.150+0000 7f996dffb700 1 -- 172.21.15.173:0/3776119932 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f99740095f0 con 0x7f99780f5410 2024-01-21T19:55:20.143 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.150+0000 7f996dffb700 1 --2- 172.21.15.173:0/3776119932 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f996402ff60 0x7f9964032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:20.143 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.150+0000 7f996dffb700 1 -- 172.21.15.173:0/3776119932 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9974041000 con 0x7f99780f5410 2024-01-21T19:55:20.144 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.150+0000 7f997cf88700 1 --2- 172.21.15.173:0/3776119932 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f996402ff60 0x7f9964032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:20.144 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.151+0000 7f997cf88700 1 --2- 172.21.15.173:0/3776119932 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f996402ff60 0x7f9964032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f99680096a0 tx=0x7f9968006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:20.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.154+0000 7f996dffb700 1 -- 172.21.15.173:0/3776119932 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9974014420 con 0x7f99780f5410 2024-01-21T19:55:20.349 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.357+0000 7f997f9ed700 1 -- 172.21.15.173:0/3776119932 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f99780fae60 con 0x7f99780f5410 2024-01-21T19:55:20.349 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.358+0000 7f996dffb700 1 -- 172.21.15.173:0/3776119932 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f997401d7f0 con 0x7f99780f5410 2024-01-21T19:55:20.350 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:20.350 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:20.351 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.360+0000 7f997f9ed700 1 -- 172.21.15.173:0/3776119932 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f996402ff60 msgr2=0x7f9964032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:20.351 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.360+0000 7f997f9ed700 1 --2- 172.21.15.173:0/3776119932 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f996402ff60 0x7f9964032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f99680096a0 tx=0x7f9968006b40).stop 2024-01-21T19:55:20.351 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.360+0000 7f997f9ed700 1 -- 172.21.15.173:0/3776119932 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99780f5410 msgr2=0x7f9978109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:20.351 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.360+0000 7f997f9ed700 1 --2- 172.21.15.173:0/3776119932 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99780f5410 0x7f9978109390 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f9974013510 tx=0x7f997400b040).stop 2024-01-21T19:55:20.351 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.360+0000 7f997f9ed700 1 -- 172.21.15.173:0/3776119932 shutdown_connections 2024-01-21T19:55:20.352 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.360+0000 7f997f9ed700 1 --2- 172.21.15.173:0/3776119932 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f996402ff60 0x7f9964032420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:20.352 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.360+0000 7f997f9ed700 1 --2- 172.21.15.173:0/3776119932 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f99780f5410 0x7f9978109390 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:20.352 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.360+0000 7f997f9ed700 1 -- 172.21.15.173:0/3776119932 >> 172.21.15.173:0/3776119932 conn(0x7f99780f0b70 msgr2=0x7f99780f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:20.352 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.360+0000 7f997f9ed700 1 -- 172.21.15.173:0/3776119932 shutdown_connections 2024-01-21T19:55:20.352 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:20.360+0000 7f997f9ed700 1 -- 172.21.15.173:0/3776119932 wait complete. 2024-01-21T19:55:20.352 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:55:21.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:20 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:18.997595+0000 mgr.smithi169.espgue (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:21.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:20 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:20.358866+0000 mon.smithi169 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.173:0/3776119932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:55:21.735 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:55:21.735 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:22.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.884+0000 7f577a4b1700 1 -- 172.21.15.173:0/1901478595 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5774060200 msgr2=0x7f57740fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:22.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.884+0000 7f577a4b1700 1 --2- 172.21.15.173:0/1901478595 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5774060200 0x7f57740fb570 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f5764009fb0 tx=0x7f5764009b30).stop 2024-01-21T19:55:22.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.884+0000 7f577a4b1700 1 -- 172.21.15.173:0/1901478595 shutdown_connections 2024-01-21T19:55:22.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.884+0000 7f577a4b1700 1 --2- 172.21.15.173:0/1901478595 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5774060200 0x7f57740fb570 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:22.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.884+0000 7f577a4b1700 1 -- 172.21.15.173:0/1901478595 >> 172.21.15.173:0/1901478595 conn(0x7f57740f0b70 msgr2=0x7f57740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:22.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.884+0000 7f577a4b1700 1 -- 172.21.15.173:0/1901478595 shutdown_connections 2024-01-21T19:55:22.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.884+0000 7f577a4b1700 1 -- 172.21.15.173:0/1901478595 wait complete. 2024-01-21T19:55:22.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.885+0000 7f577a4b1700 1 Processor -- start 2024-01-21T19:55:22.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.885+0000 7f577a4b1700 1 -- start start 2024-01-21T19:55:22.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.885+0000 7f577a4b1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5774060200 0x7f57741088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:22.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.885+0000 7f577a4b1700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5774108dc0 con 0x7f5774060200 2024-01-21T19:55:22.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.885+0000 7f5773fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5774060200 0x7f57741088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:22.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.885+0000 7f5773fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5774060200 0x7f57741088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:43918/0 (socket says 172.21.15.173:43918) 2024-01-21T19:55:22.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.885+0000 7f5773fff700 1 -- 172.21.15.173:0/1811315674 learned_addr learned my addr 172.21.15.173:0/1811315674 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:22.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.885+0000 7f5773fff700 1 -- 172.21.15.173:0/1811315674 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5764005040 con 0x7f5774060200 2024-01-21T19:55:22.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.886+0000 7f5773fff700 1 --2- 172.21.15.173:0/1811315674 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5774060200 0x7f57741088d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f5764013510 tx=0x7f576400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:22.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.886+0000 7f5770ff9700 1 -- 172.21.15.173:0/1811315674 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5764009320 con 0x7f5774060200 2024-01-21T19:55:22.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.886+0000 7f577a4b1700 1 -- 172.21.15.173:0/1811315674 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5774108fc0 con 0x7f5774060200 2024-01-21T19:55:22.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.886+0000 7f5770ff9700 1 -- 172.21.15.173:0/1811315674 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5764013f00 con 0x7f5774060200 2024-01-21T19:55:22.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.886+0000 7f5770ff9700 1 -- 172.21.15.173:0/1811315674 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5764014dc0 con 0x7f5774060200 2024-01-21T19:55:22.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.886+0000 7f577a4b1700 1 -- 172.21.15.173:0/1811315674 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5774109450 con 0x7f5774060200 2024-01-21T19:55:22.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.887+0000 7f5770ff9700 1 -- 172.21.15.173:0/1811315674 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5764014420 con 0x7f5774060200 2024-01-21T19:55:22.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.887+0000 7f577a4b1700 1 -- 172.21.15.173:0/1811315674 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5774045ad0 con 0x7f5774060200 2024-01-21T19:55:22.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.887+0000 7f5770ff9700 1 --2- 172.21.15.173:0/1811315674 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f575c02ffb0 0x7f575c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:22.880 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.887+0000 7f5770ff9700 1 -- 172.21.15.173:0/1811315674 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5764040fd0 con 0x7f5774060200 2024-01-21T19:55:22.880 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.887+0000 7f57737fe700 1 --2- 172.21.15.173:0/1811315674 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f575c02ffb0 0x7f575c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:22.880 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.887+0000 7f57737fe700 1 --2- 172.21.15.173:0/1811315674 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f575c02ffb0 0x7f575c032470 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f57680096a0 tx=0x7f5768006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:22.883 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:22.891+0000 7f5770ff9700 1 -- 172.21.15.173:0/1811315674 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f576406d2b0 con 0x7f5774060200 2024-01-21T19:55:23.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:23.098+0000 7f577a4b1700 1 -- 172.21.15.173:0/1811315674 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f577410a290 con 0x7f5774060200 2024-01-21T19:55:23.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:23.099+0000 7f5770ff9700 1 -- 172.21.15.173:0/1811315674 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f57640095f0 con 0x7f5774060200 2024-01-21T19:55:23.091 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:23.091 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:23.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:23.101+0000 7f577a4b1700 1 -- 172.21.15.173:0/1811315674 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f575c02ffb0 msgr2=0x7f575c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:23.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:23.101+0000 7f577a4b1700 1 --2- 172.21.15.173:0/1811315674 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f575c02ffb0 0x7f575c032470 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f57680096a0 tx=0x7f5768006b40).stop 2024-01-21T19:55:23.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:23.101+0000 7f577a4b1700 1 -- 172.21.15.173:0/1811315674 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5774060200 msgr2=0x7f57741088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:23.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:23.101+0000 7f577a4b1700 1 --2- 172.21.15.173:0/1811315674 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5774060200 0x7f57741088d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f5764013510 tx=0x7f576400b040).stop 2024-01-21T19:55:23.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:23.101+0000 7f577a4b1700 1 -- 172.21.15.173:0/1811315674 shutdown_connections 2024-01-21T19:55:23.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:23.101+0000 7f577a4b1700 1 --2- 172.21.15.173:0/1811315674 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f575c02ffb0 0x7f575c032470 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:23.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:23.101+0000 7f577a4b1700 1 --2- 172.21.15.173:0/1811315674 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5774060200 0x7f57741088d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:23.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:23.101+0000 7f577a4b1700 1 -- 172.21.15.173:0/1811315674 >> 172.21.15.173:0/1811315674 conn(0x7f57740f0b70 msgr2=0x7f57740f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:23.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:23.101+0000 7f577a4b1700 1 -- 172.21.15.173:0/1811315674 shutdown_connections 2024-01-21T19:55:23.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:23.101+0000 7f577a4b1700 1 -- 172.21.15.173:0/1811315674 wait complete. 2024-01-21T19:55:23.093 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:55:23.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:22 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:20.997974+0000 mgr.smithi169.espgue (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:24.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:23 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:23.099807+0000 mon.smithi169 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.173:0/1811315674' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:55:24.489 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:55:24.490 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:25.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:24 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:22.998391+0000 mgr.smithi169.espgue (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:25.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.617+0000 7f4a2bd86700 1 -- 172.21.15.173:0/2393475793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a240f5410 msgr2=0x7f4a240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:25.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.617+0000 7f4a2bd86700 1 --2- 172.21.15.173:0/2393475793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a240f5410 0x7f4a240f5830 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f4a14009fb0 tx=0x7f4a14009b30).stop 2024-01-21T19:55:25.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.617+0000 7f4a2bd86700 1 -- 172.21.15.173:0/2393475793 shutdown_connections 2024-01-21T19:55:25.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.617+0000 7f4a2bd86700 1 --2- 172.21.15.173:0/2393475793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a240f5410 0x7f4a240f5830 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:25.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.617+0000 7f4a2bd86700 1 -- 172.21.15.173:0/2393475793 >> 172.21.15.173:0/2393475793 conn(0x7f4a240f0b70 msgr2=0x7f4a240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:25.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.617+0000 7f4a2bd86700 1 -- 172.21.15.173:0/2393475793 shutdown_connections 2024-01-21T19:55:25.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.617+0000 7f4a2bd86700 1 -- 172.21.15.173:0/2393475793 wait complete. 2024-01-21T19:55:25.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.618+0000 7f4a2bd86700 1 Processor -- start 2024-01-21T19:55:25.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.618+0000 7f4a2bd86700 1 -- start start 2024-01-21T19:55:25.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.618+0000 7f4a2bd86700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a240f5410 0x7f4a24109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:25.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.618+0000 7f4a2bd86700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a24060240 con 0x7f4a240f5410 2024-01-21T19:55:25.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.618+0000 7f4a29b22700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a240f5410 0x7f4a24109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:25.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.618+0000 7f4a29b22700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a240f5410 0x7f4a24109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:33310/0 (socket says 172.21.15.173:33310) 2024-01-21T19:55:25.611 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.618+0000 7f4a29b22700 1 -- 172.21.15.173:0/862639508 learned_addr learned my addr 172.21.15.173:0/862639508 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:25.612 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.619+0000 7f4a29b22700 1 -- 172.21.15.173:0/862639508 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a14005040 con 0x7f4a240f5410 2024-01-21T19:55:25.612 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.619+0000 7f4a29b22700 1 --2- 172.21.15.173:0/862639508 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a240f5410 0x7f4a24109430 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f4a14013510 tx=0x7f4a1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:25.612 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.619+0000 7f4a1a7fc700 1 -- 172.21.15.173:0/862639508 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a14009320 con 0x7f4a240f5410 2024-01-21T19:55:25.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.619+0000 7f4a1a7fc700 1 -- 172.21.15.173:0/862639508 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a14013f00 con 0x7f4a240f5410 2024-01-21T19:55:25.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.619+0000 7f4a2bd86700 1 -- 172.21.15.173:0/862639508 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a24060440 con 0x7f4a240f5410 2024-01-21T19:55:25.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.619+0000 7f4a1a7fc700 1 -- 172.21.15.173:0/862639508 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a14014d90 con 0x7f4a240f5410 2024-01-21T19:55:25.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.619+0000 7f4a2bd86700 1 -- 172.21.15.173:0/862639508 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a24109ae0 con 0x7f4a240f5410 2024-01-21T19:55:25.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.620+0000 7f4a1a7fc700 1 -- 172.21.15.173:0/862639508 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a140263a0 con 0x7f4a240f5410 2024-01-21T19:55:25.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.620+0000 7f4a2bd86700 1 -- 172.21.15.173:0/862639508 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a24045ad0 con 0x7f4a240f5410 2024-01-21T19:55:25.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.620+0000 7f4a1a7fc700 1 --2- 172.21.15.173:0/862639508 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4a1002ff60 0x7f4a10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:25.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.620+0000 7f4a1a7fc700 1 -- 172.21.15.173:0/862639508 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a14041300 con 0x7f4a240f5410 2024-01-21T19:55:25.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.620+0000 7f4a29321700 1 --2- 172.21.15.173:0/862639508 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4a1002ff60 0x7f4a10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:25.613 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.621+0000 7f4a29321700 1 --2- 172.21.15.173:0/862639508 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4a1002ff60 0x7f4a10032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f4a200096a0 tx=0x7f4a20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:25.616 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.625+0000 7f4a1a7fc700 1 -- 172.21.15.173:0/862639508 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a140126f0 con 0x7f4a240f5410 2024-01-21T19:55:25.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.830+0000 7f4a2bd86700 1 -- 172.21.15.173:0/862639508 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a240fae60 con 0x7f4a240f5410 2024-01-21T19:55:25.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.831+0000 7f4a1a7fc700 1 -- 172.21.15.173:0/862639508 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4a14012330 con 0x7f4a240f5410 2024-01-21T19:55:25.823 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:25.823 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:25.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.832+0000 7f4a2bd86700 1 -- 172.21.15.173:0/862639508 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4a1002ff60 msgr2=0x7f4a10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:25.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.832+0000 7f4a2bd86700 1 --2- 172.21.15.173:0/862639508 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4a1002ff60 0x7f4a10032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f4a200096a0 tx=0x7f4a20006b40).stop 2024-01-21T19:55:25.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.832+0000 7f4a2bd86700 1 -- 172.21.15.173:0/862639508 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a240f5410 msgr2=0x7f4a24109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:25.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.832+0000 7f4a2bd86700 1 --2- 172.21.15.173:0/862639508 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a240f5410 0x7f4a24109430 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f4a14013510 tx=0x7f4a1400b040).stop 2024-01-21T19:55:25.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.833+0000 7f4a2bd86700 1 -- 172.21.15.173:0/862639508 shutdown_connections 2024-01-21T19:55:25.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.833+0000 7f4a2bd86700 1 --2- 172.21.15.173:0/862639508 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4a1002ff60 0x7f4a10032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:25.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.833+0000 7f4a2bd86700 1 --2- 172.21.15.173:0/862639508 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4a240f5410 0x7f4a24109430 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:25.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.833+0000 7f4a2bd86700 1 -- 172.21.15.173:0/862639508 >> 172.21.15.173:0/862639508 conn(0x7f4a240f0b70 msgr2=0x7f4a240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:25.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.833+0000 7f4a2bd86700 1 -- 172.21.15.173:0/862639508 shutdown_connections 2024-01-21T19:55:25.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:25.833+0000 7f4a2bd86700 1 -- 172.21.15.173:0/862639508 wait complete. 2024-01-21T19:55:25.825 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:55:27.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:26 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:24.998792+0000 mgr.smithi169.espgue (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:27.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:26 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:25.831615+0000 mon.smithi169 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.173:0/862639508' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:55:27.242 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:55:27.242 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:28.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.437+0000 7f1354df8700 1 -- 172.21.15.173:0/3952050321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13500f5410 msgr2=0x7f13500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:28.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.437+0000 7f1354df8700 1 --2- 172.21.15.173:0/3952050321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13500f5410 0x7f13500f5830 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f1338009fb0 tx=0x7f1338009b30).stop 2024-01-21T19:55:28.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.438+0000 7f1354df8700 1 -- 172.21.15.173:0/3952050321 shutdown_connections 2024-01-21T19:55:28.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.438+0000 7f1354df8700 1 --2- 172.21.15.173:0/3952050321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13500f5410 0x7f13500f5830 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:28.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.438+0000 7f1354df8700 1 -- 172.21.15.173:0/3952050321 >> 172.21.15.173:0/3952050321 conn(0x7f13500f0b70 msgr2=0x7f13500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:28.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.438+0000 7f1354df8700 1 -- 172.21.15.173:0/3952050321 shutdown_connections 2024-01-21T19:55:28.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.438+0000 7f1354df8700 1 -- 172.21.15.173:0/3952050321 wait complete. 2024-01-21T19:55:28.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.438+0000 7f1354df8700 1 Processor -- start 2024-01-21T19:55:28.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.438+0000 7f1354df8700 1 -- start start 2024-01-21T19:55:28.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.438+0000 7f1354df8700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13500f5410 0x7f13500fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:28.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.438+0000 7f1354df8700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13500fb820 con 0x7f13500f5410 2024-01-21T19:55:28.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.438+0000 7f134e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13500f5410 0x7f13500fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:28.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.438+0000 7f134e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13500f5410 0x7f13500fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:33318/0 (socket says 172.21.15.173:33318) 2024-01-21T19:55:28.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.438+0000 7f134e59c700 1 -- 172.21.15.173:0/3857116129 learned_addr learned my addr 172.21.15.173:0/3857116129 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:28.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.439+0000 7f134e59c700 1 -- 172.21.15.173:0/3857116129 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1338005040 con 0x7f13500f5410 2024-01-21T19:55:28.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.439+0000 7f134e59c700 1 --2- 172.21.15.173:0/3857116129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13500f5410 0x7f13500fd3a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f1338013510 tx=0x7f133800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:28.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.439+0000 7f1346ffd700 1 -- 172.21.15.173:0/3857116129 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1338009320 con 0x7f13500f5410 2024-01-21T19:55:28.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.439+0000 7f1346ffd700 1 -- 172.21.15.173:0/3857116129 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1338013f00 con 0x7f13500f5410 2024-01-21T19:55:28.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.439+0000 7f1354df8700 1 -- 172.21.15.173:0/3857116129 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13500fb9c0 con 0x7f13500f5410 2024-01-21T19:55:28.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.439+0000 7f1346ffd700 1 -- 172.21.15.173:0/3857116129 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1338014d90 con 0x7f13500f5410 2024-01-21T19:55:28.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.439+0000 7f1354df8700 1 -- 172.21.15.173:0/3857116129 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13500fbe50 con 0x7f13500f5410 2024-01-21T19:55:28.433 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.440+0000 7f1346ffd700 1 -- 172.21.15.173:0/3857116129 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1338014420 con 0x7f13500f5410 2024-01-21T19:55:28.433 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.440+0000 7f1354df8700 1 -- 172.21.15.173:0/3857116129 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1350059100 con 0x7f13500f5410 2024-01-21T19:55:28.433 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.440+0000 7f1346ffd700 1 --2- 172.21.15.173:0/3857116129 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f133c02ff60 0x7f133c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:28.433 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.440+0000 7f1346ffd700 1 -- 172.21.15.173:0/3857116129 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1338041000 con 0x7f13500f5410 2024-01-21T19:55:28.433 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.440+0000 7f134dd9b700 1 --2- 172.21.15.173:0/3857116129 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f133c02ff60 0x7f133c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:28.433 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.441+0000 7f134dd9b700 1 --2- 172.21.15.173:0/3857116129 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f133c02ff60 0x7f133c032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f13400096a0 tx=0x7f1340006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:28.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.445+0000 7f1346ffd700 1 -- 172.21.15.173:0/3857116129 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f13380095f0 con 0x7f13500f5410 2024-01-21T19:55:28.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.651+0000 7f1354df8700 1 -- 172.21.15.173:0/3857116129 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13500fae60 con 0x7f13500f5410 2024-01-21T19:55:28.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.651+0000 7f1346ffd700 1 -- 172.21.15.173:0/3857116129 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f133801db70 con 0x7f13500f5410 2024-01-21T19:55:28.643 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:28.643 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:28.644 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.653+0000 7f1354df8700 1 -- 172.21.15.173:0/3857116129 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f133c02ff60 msgr2=0x7f133c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:28.644 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.653+0000 7f1354df8700 1 --2- 172.21.15.173:0/3857116129 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f133c02ff60 0x7f133c032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f13400096a0 tx=0x7f1340006b40).stop 2024-01-21T19:55:28.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.653+0000 7f1354df8700 1 -- 172.21.15.173:0/3857116129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13500f5410 msgr2=0x7f13500fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:28.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.653+0000 7f1354df8700 1 --2- 172.21.15.173:0/3857116129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13500f5410 0x7f13500fd3a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f1338013510 tx=0x7f133800b040).stop 2024-01-21T19:55:28.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.653+0000 7f1354df8700 1 -- 172.21.15.173:0/3857116129 shutdown_connections 2024-01-21T19:55:28.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.653+0000 7f1354df8700 1 --2- 172.21.15.173:0/3857116129 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f133c02ff60 0x7f133c032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:28.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.653+0000 7f1354df8700 1 --2- 172.21.15.173:0/3857116129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13500f5410 0x7f13500fd3a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:28.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.653+0000 7f1354df8700 1 -- 172.21.15.173:0/3857116129 >> 172.21.15.173:0/3857116129 conn(0x7f13500f0b70 msgr2=0x7f135005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:28.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.653+0000 7f1354df8700 1 -- 172.21.15.173:0/3857116129 shutdown_connections 2024-01-21T19:55:28.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:28.653+0000 7f1354df8700 1 -- 172.21.15.173:0/3857116129 wait complete. 2024-01-21T19:55:28.645 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:55:29.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:28 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:26.999140+0000 mgr.smithi169.espgue (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:29.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:28 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:28.652169+0000 mon.smithi169 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.173:0/3857116129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:55:30.035 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:55:30.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:31.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:30 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:28.999447+0000 mgr.smithi169.espgue (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:31.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.180+0000 7f5424629700 1 -- 172.21.15.173:0/1423126520 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f541c0601e0 msgr2=0x7f541c0f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:31.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.180+0000 7f5424629700 1 --2- 172.21.15.173:0/1423126520 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f541c0601e0 0x7f541c0f9190 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f5410009fb0 tx=0x7f5410009b30).stop 2024-01-21T19:55:31.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.180+0000 7f5424629700 1 -- 172.21.15.173:0/1423126520 shutdown_connections 2024-01-21T19:55:31.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.180+0000 7f5424629700 1 --2- 172.21.15.173:0/1423126520 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f541c0601e0 0x7f541c0f9190 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:31.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.180+0000 7f5424629700 1 -- 172.21.15.173:0/1423126520 >> 172.21.15.173:0/1423126520 conn(0x7f541c0f0b70 msgr2=0x7f541c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:31.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.180+0000 7f5424629700 1 -- 172.21.15.173:0/1423126520 shutdown_connections 2024-01-21T19:55:31.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.180+0000 7f5424629700 1 -- 172.21.15.173:0/1423126520 wait complete. 2024-01-21T19:55:31.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.181+0000 7f5424629700 1 Processor -- start 2024-01-21T19:55:31.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.181+0000 7f5424629700 1 -- start start 2024-01-21T19:55:31.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.181+0000 7f5424629700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f541c0601e0 0x7f541c0f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:31.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.181+0000 7f5424629700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0f5090 con 0x7f541c0601e0 2024-01-21T19:55:31.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.181+0000 7f54223c5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f541c0601e0 0x7f541c0f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:31.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.181+0000 7f54223c5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f541c0601e0 0x7f541c0f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:33344/0 (socket says 172.21.15.173:33344) 2024-01-21T19:55:31.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.181+0000 7f54223c5700 1 -- 172.21.15.173:0/3010944804 learned_addr learned my addr 172.21.15.173:0/3010944804 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:31.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.181+0000 7f54223c5700 1 -- 172.21.15.173:0/3010944804 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5410005040 con 0x7f541c0601e0 2024-01-21T19:55:31.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.181+0000 7f54223c5700 1 --2- 172.21.15.173:0/3010944804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f541c0601e0 0x7f541c0f6c10 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f5410004dd0 tx=0x7f5410009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:31.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.182+0000 7f540effd700 1 -- 172.21.15.173:0/3010944804 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5410018070 con 0x7f541c0601e0 2024-01-21T19:55:31.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.182+0000 7f540effd700 1 -- 172.21.15.173:0/3010944804 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5410013410 con 0x7f541c0601e0 2024-01-21T19:55:31.175 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.182+0000 7f5424629700 1 -- 172.21.15.173:0/3010944804 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f541c0f5230 con 0x7f541c0601e0 2024-01-21T19:55:31.175 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.182+0000 7f540effd700 1 -- 172.21.15.173:0/3010944804 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f541001c3c0 con 0x7f541c0601e0 2024-01-21T19:55:31.175 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.182+0000 7f5424629700 1 -- 172.21.15.173:0/3010944804 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f541c0f56c0 con 0x7f541c0601e0 2024-01-21T19:55:31.175 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.183+0000 7f540effd700 1 -- 172.21.15.173:0/3010944804 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f541001d420 con 0x7f541c0601e0 2024-01-21T19:55:31.176 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.183+0000 7f5424629700 1 -- 172.21.15.173:0/3010944804 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f541c059100 con 0x7f541c0601e0 2024-01-21T19:55:31.176 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.183+0000 7f540effd700 1 --2- 172.21.15.173:0/3010944804 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f540802ff60 0x7f5408032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:31.176 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.183+0000 7f540effd700 1 -- 172.21.15.173:0/3010944804 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5410041310 con 0x7f541c0601e0 2024-01-21T19:55:31.176 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.183+0000 7f5421bc4700 1 --2- 172.21.15.173:0/3010944804 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f540802ff60 0x7f5408032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:31.176 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.183+0000 7f5421bc4700 1 --2- 172.21.15.173:0/3010944804 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f540802ff60 0x7f5408032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f54180096a0 tx=0x7f5418006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:31.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.187+0000 7f540effd700 1 -- 172.21.15.173:0/3010944804 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f541001d820 con 0x7f541c0601e0 2024-01-21T19:55:31.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.396+0000 7f5424629700 1 -- 172.21.15.173:0/3010944804 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f541c0fbf60 con 0x7f541c0601e0 2024-01-21T19:55:31.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.397+0000 7f540effd700 1 -- 172.21.15.173:0/3010944804 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f541001b370 con 0x7f541c0601e0 2024-01-21T19:55:31.388 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:31.389 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:31.390 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.398+0000 7f5424629700 1 -- 172.21.15.173:0/3010944804 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f540802ff60 msgr2=0x7f5408032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:31.390 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.398+0000 7f5424629700 1 --2- 172.21.15.173:0/3010944804 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f540802ff60 0x7f5408032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f54180096a0 tx=0x7f5418006b40).stop 2024-01-21T19:55:31.390 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.398+0000 7f5424629700 1 -- 172.21.15.173:0/3010944804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f541c0601e0 msgr2=0x7f541c0f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:31.390 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.398+0000 7f5424629700 1 --2- 172.21.15.173:0/3010944804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f541c0601e0 0x7f541c0f6c10 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f5410004dd0 tx=0x7f5410009b30).stop 2024-01-21T19:55:31.390 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.398+0000 7f5424629700 1 -- 172.21.15.173:0/3010944804 shutdown_connections 2024-01-21T19:55:31.390 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.398+0000 7f5424629700 1 --2- 172.21.15.173:0/3010944804 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f540802ff60 0x7f5408032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:31.390 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.398+0000 7f5424629700 1 --2- 172.21.15.173:0/3010944804 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f541c0601e0 0x7f541c0f6c10 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:31.390 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.398+0000 7f5424629700 1 -- 172.21.15.173:0/3010944804 >> 172.21.15.173:0/3010944804 conn(0x7f541c0f0b70 msgr2=0x7f541c0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:31.390 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.398+0000 7f5424629700 1 -- 172.21.15.173:0/3010944804 shutdown_connections 2024-01-21T19:55:31.390 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:31.398+0000 7f5424629700 1 -- 172.21.15.173:0/3010944804 wait complete. 2024-01-21T19:55:31.391 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:55:32.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:31.076059+0000 mon.smithi169 (mon.0) 276 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/mirror_snapshot_schedule"}]: dispatch 2024-01-21T19:55:32.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:31.079242+0000 mon.smithi169 (mon.0) 277 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/trash_purge_schedule"}]: dispatch 2024-01-21T19:55:32.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:31.397607+0000 mon.smithi169 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.173:0/3010944804' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:55:32.792 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:55:32.792 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:33.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:32 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:30.999827+0000 mgr.smithi169.espgue (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:34.001 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.010+0000 7f9212e5d700 1 -- 172.21.15.173:0/3019017757 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f920c0f6480 msgr2=0x7f920c0f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:34.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.010+0000 7f9212e5d700 1 --2- 172.21.15.173:0/3019017757 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f920c0f6480 0x7f920c0f68a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f9200009fb0 tx=0x7f9200009b30).stop 2024-01-21T19:55:34.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.010+0000 7f9212e5d700 1 -- 172.21.15.173:0/3019017757 shutdown_connections 2024-01-21T19:55:34.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.010+0000 7f9212e5d700 1 --2- 172.21.15.173:0/3019017757 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f920c0f6480 0x7f920c0f68a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:34.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.010+0000 7f9212e5d700 1 -- 172.21.15.173:0/3019017757 >> 172.21.15.173:0/3019017757 conn(0x7f920c0f1c00 msgr2=0x7f920c0f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:34.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.010+0000 7f9212e5d700 1 -- 172.21.15.173:0/3019017757 shutdown_connections 2024-01-21T19:55:34.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.010+0000 7f9212e5d700 1 -- 172.21.15.173:0/3019017757 wait complete. 2024-01-21T19:55:34.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.011+0000 7f9212e5d700 1 Processor -- start 2024-01-21T19:55:34.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.011+0000 7f9212e5d700 1 -- start start 2024-01-21T19:55:34.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.011+0000 7f9212e5d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f920c0f6480 0x7f920c1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:34.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.011+0000 7f9212e5d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920c106bc0 con 0x7f920c0f6480 2024-01-21T19:55:34.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.011+0000 7f9210bf9700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f920c0f6480 0x7f920c1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:34.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.011+0000 7f9210bf9700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f920c0f6480 0x7f920c1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:38138/0 (socket says 172.21.15.173:38138) 2024-01-21T19:55:34.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.011+0000 7f9210bf9700 1 -- 172.21.15.173:0/4178292269 learned_addr learned my addr 172.21.15.173:0/4178292269 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:34.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.012+0000 7f9210bf9700 1 -- 172.21.15.173:0/4178292269 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9200005040 con 0x7f920c0f6480 2024-01-21T19:55:34.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.012+0000 7f9210bf9700 1 --2- 172.21.15.173:0/4178292269 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f920c0f6480 0x7f920c1066d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f9200013510 tx=0x7f920000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:34.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.012+0000 7f92097fa700 1 -- 172.21.15.173:0/4178292269 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9200009320 con 0x7f920c0f6480 2024-01-21T19:55:34.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.012+0000 7f92097fa700 1 -- 172.21.15.173:0/4178292269 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9200013f00 con 0x7f920c0f6480 2024-01-21T19:55:34.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.012+0000 7f9212e5d700 1 -- 172.21.15.173:0/4178292269 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f920c106dc0 con 0x7f920c0f6480 2024-01-21T19:55:34.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.012+0000 7f92097fa700 1 -- 172.21.15.173:0/4178292269 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9200014d90 con 0x7f920c0f6480 2024-01-21T19:55:34.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.012+0000 7f9212e5d700 1 -- 172.21.15.173:0/4178292269 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f920c107250 con 0x7f920c0f6480 2024-01-21T19:55:34.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.013+0000 7f92097fa700 1 -- 172.21.15.173:0/4178292269 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f92000263a0 con 0x7f920c0f6480 2024-01-21T19:55:34.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.013+0000 7f92097fa700 1 --2- 172.21.15.173:0/4178292269 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f91fc02ff60 0x7f91fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:34.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.013+0000 7f92097fa700 1 -- 172.21.15.173:0/4178292269 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9200041390 con 0x7f920c0f6480 2024-01-21T19:55:34.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.013+0000 7f9212e5d700 1 -- 172.21.15.173:0/4178292269 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f920c045ad0 con 0x7f920c0f6480 2024-01-21T19:55:34.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.013+0000 7f920bfff700 1 --2- 172.21.15.173:0/4178292269 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f91fc02ff60 0x7f91fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:34.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.014+0000 7f920bfff700 1 --2- 172.21.15.173:0/4178292269 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f91fc02ff60 0x7f91fc032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f91f80096a0 tx=0x7f91f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:34.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.018+0000 7f92097fa700 1 -- 172.21.15.173:0/4178292269 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f920001cb10 con 0x7f920c0f6480 2024-01-21T19:55:34.214 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.223+0000 7f9212e5d700 1 -- 172.21.15.173:0/4178292269 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f920c108100 con 0x7f920c0f6480 2024-01-21T19:55:34.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.223+0000 7f92097fa700 1 -- 172.21.15.173:0/4178292269 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f920002b030 con 0x7f920c0f6480 2024-01-21T19:55:34.215 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:34.215 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:34.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.225+0000 7f9212e5d700 1 -- 172.21.15.173:0/4178292269 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f91fc02ff60 msgr2=0x7f91fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:34.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.225+0000 7f9212e5d700 1 --2- 172.21.15.173:0/4178292269 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f91fc02ff60 0x7f91fc032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f91f80096a0 tx=0x7f91f8006b40).stop 2024-01-21T19:55:34.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.225+0000 7f9212e5d700 1 -- 172.21.15.173:0/4178292269 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f920c0f6480 msgr2=0x7f920c1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:34.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.225+0000 7f9212e5d700 1 --2- 172.21.15.173:0/4178292269 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f920c0f6480 0x7f920c1066d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f9200013510 tx=0x7f920000b040).stop 2024-01-21T19:55:34.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.225+0000 7f9212e5d700 1 -- 172.21.15.173:0/4178292269 shutdown_connections 2024-01-21T19:55:34.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.225+0000 7f9212e5d700 1 --2- 172.21.15.173:0/4178292269 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f91fc02ff60 0x7f91fc032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:34.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.225+0000 7f9212e5d700 1 --2- 172.21.15.173:0/4178292269 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f920c0f6480 0x7f920c1066d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:34.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.225+0000 7f9212e5d700 1 -- 172.21.15.173:0/4178292269 >> 172.21.15.173:0/4178292269 conn(0x7f920c0f1c00 msgr2=0x7f920c0f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:34.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.225+0000 7f9212e5d700 1 -- 172.21.15.173:0/4178292269 shutdown_connections 2024-01-21T19:55:34.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:34.225+0000 7f9212e5d700 1 -- 172.21.15.173:0/4178292269 wait complete. 2024-01-21T19:55:34.218 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:55:35.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:34 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:33.000219+0000 mgr.smithi169.espgue (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:35.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:34 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:34.224209+0000 mon.smithi169 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.173:0/4178292269' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:55:35.594 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:55:35.595 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:36.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.811+0000 7f3dc1278700 1 -- 172.21.15.173:0/1706112662 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3dbc0f51a0 msgr2=0x7f3dbc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:36.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.811+0000 7f3dc1278700 1 --2- 172.21.15.173:0/1706112662 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3dbc0f51a0 0x7f3dbc0f55c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f3dac009fb0 tx=0x7f3dac009b30).stop 2024-01-21T19:55:36.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.812+0000 7f3dc1278700 1 -- 172.21.15.173:0/1706112662 shutdown_connections 2024-01-21T19:55:36.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.812+0000 7f3dc1278700 1 --2- 172.21.15.173:0/1706112662 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3dbc0f51a0 0x7f3dbc0f55c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:36.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.812+0000 7f3dc1278700 1 -- 172.21.15.173:0/1706112662 >> 172.21.15.173:0/1706112662 conn(0x7f3dbc0f09a0 msgr2=0x7f3dbc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:36.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.812+0000 7f3dc1278700 1 -- 172.21.15.173:0/1706112662 shutdown_connections 2024-01-21T19:55:36.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.812+0000 7f3dc1278700 1 -- 172.21.15.173:0/1706112662 wait complete. 2024-01-21T19:55:36.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.812+0000 7f3dc1278700 1 Processor -- start 2024-01-21T19:55:36.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.812+0000 7f3dc1278700 1 -- start start 2024-01-21T19:55:36.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.812+0000 7f3dc1278700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3dbc0f51a0 0x7f3dbc1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:36.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.812+0000 7f3dc1278700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dbc060220 con 0x7f3dbc0f51a0 2024-01-21T19:55:36.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.813+0000 7f3dbbfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3dbc0f51a0 0x7f3dbc1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:36.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.813+0000 7f3dbbfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3dbc0f51a0 0x7f3dbc1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:38156/0 (socket says 172.21.15.173:38156) 2024-01-21T19:55:36.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.813+0000 7f3dbbfff700 1 -- 172.21.15.173:0/869129287 learned_addr learned my addr 172.21.15.173:0/869129287 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:36.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.813+0000 7f3dbbfff700 1 -- 172.21.15.173:0/869129287 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dac005040 con 0x7f3dbc0f51a0 2024-01-21T19:55:36.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.813+0000 7f3dbbfff700 1 --2- 172.21.15.173:0/869129287 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3dbc0f51a0 0x7f3dbc1091d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f3dac013510 tx=0x7f3dac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:36.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.813+0000 7f3db8ff9700 1 -- 172.21.15.173:0/869129287 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dac009320 con 0x7f3dbc0f51a0 2024-01-21T19:55:36.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.813+0000 7f3dc1278700 1 -- 172.21.15.173:0/869129287 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dbc060420 con 0x7f3dbc0f51a0 2024-01-21T19:55:36.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.813+0000 7f3db8ff9700 1 -- 172.21.15.173:0/869129287 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3dac013f00 con 0x7f3dbc0f51a0 2024-01-21T19:55:36.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.813+0000 7f3db8ff9700 1 -- 172.21.15.173:0/869129287 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dac014dc0 con 0x7f3dbc0f51a0 2024-01-21T19:55:36.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.813+0000 7f3dc1278700 1 -- 172.21.15.173:0/869129287 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dbc109880 con 0x7f3dbc0f51a0 2024-01-21T19:55:36.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.814+0000 7f3db8ff9700 1 -- 172.21.15.173:0/869129287 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3dac0095f0 con 0x7f3dbc0f51a0 2024-01-21T19:55:36.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.814+0000 7f3dc1278700 1 -- 172.21.15.173:0/869129287 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dbc059100 con 0x7f3dbc0f51a0 2024-01-21T19:55:36.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.814+0000 7f3db8ff9700 1 --2- 172.21.15.173:0/869129287 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3da402ff60 0x7f3da4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:36.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.814+0000 7f3db8ff9700 1 -- 172.21.15.173:0/869129287 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3dac041000 con 0x7f3dbc0f51a0 2024-01-21T19:55:36.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.815+0000 7f3dbb7fe700 1 --2- 172.21.15.173:0/869129287 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3da402ff60 0x7f3da4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:36.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.815+0000 7f3dbb7fe700 1 --2- 172.21.15.173:0/869129287 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3da402ff60 0x7f3da4032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f3db00096a0 tx=0x7f3db0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:36.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:36.819+0000 7f3db8ff9700 1 -- 172.21.15.173:0/869129287 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3dac014420 con 0x7f3dbc0f51a0 2024-01-21T19:55:37.015 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:37.023+0000 7f3dc1278700 1 -- 172.21.15.173:0/869129287 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3dbc045ad0 con 0x7f3dbc0f51a0 2024-01-21T19:55:37.015 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:37.024+0000 7f3db8ff9700 1 -- 172.21.15.173:0/869129287 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3dac010230 con 0x7f3dbc0f51a0 2024-01-21T19:55:37.015 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:37.016 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:37.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:37.026+0000 7f3dc1278700 1 -- 172.21.15.173:0/869129287 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3da402ff60 msgr2=0x7f3da4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:37.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:37.026+0000 7f3dc1278700 1 --2- 172.21.15.173:0/869129287 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3da402ff60 0x7f3da4032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f3db00096a0 tx=0x7f3db0006b40).stop 2024-01-21T19:55:37.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:37.026+0000 7f3dc1278700 1 -- 172.21.15.173:0/869129287 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3dbc0f51a0 msgr2=0x7f3dbc1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:37.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:37.026+0000 7f3dc1278700 1 --2- 172.21.15.173:0/869129287 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3dbc0f51a0 0x7f3dbc1091d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f3dac013510 tx=0x7f3dac00b040).stop 2024-01-21T19:55:37.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:37.026+0000 7f3dc1278700 1 -- 172.21.15.173:0/869129287 shutdown_connections 2024-01-21T19:55:37.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:37.026+0000 7f3dc1278700 1 --2- 172.21.15.173:0/869129287 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3da402ff60 0x7f3da4032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:37.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:37.026+0000 7f3dc1278700 1 --2- 172.21.15.173:0/869129287 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3dbc0f51a0 0x7f3dbc1091d0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:37.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:37.026+0000 7f3dc1278700 1 -- 172.21.15.173:0/869129287 >> 172.21.15.173:0/869129287 conn(0x7f3dbc0f09a0 msgr2=0x7f3dbc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:37.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:37.026+0000 7f3dc1278700 1 -- 172.21.15.173:0/869129287 shutdown_connections 2024-01-21T19:55:37.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:37.026+0000 7f3dc1278700 1 -- 172.21.15.173:0/869129287 wait complete. 2024-01-21T19:55:37.018 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:55:37.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:36 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:35.000693+0000 mgr.smithi169.espgue (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:38.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:37 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:37.024743+0000 mon.smithi169 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.173:0/869129287' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:55:38.419 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:55:38.419 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:39.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:37.001136+0000 mgr.smithi169.espgue (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:39.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.607+0000 7f8efa3ed700 1 -- 172.21.15.173:0/3484953343 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ef40f5410 msgr2=0x7f8ef40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:39.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.607+0000 7f8efa3ed700 1 --2- 172.21.15.173:0/3484953343 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ef40f5410 0x7f8ef40f5830 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f8ee4009fb0 tx=0x7f8ee4009b30).stop 2024-01-21T19:55:39.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.607+0000 7f8efa3ed700 1 -- 172.21.15.173:0/3484953343 shutdown_connections 2024-01-21T19:55:39.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.607+0000 7f8efa3ed700 1 --2- 172.21.15.173:0/3484953343 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ef40f5410 0x7f8ef40f5830 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:39.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.607+0000 7f8efa3ed700 1 -- 172.21.15.173:0/3484953343 >> 172.21.15.173:0/3484953343 conn(0x7f8ef40f0b70 msgr2=0x7f8ef40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:39.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.607+0000 7f8efa3ed700 1 -- 172.21.15.173:0/3484953343 shutdown_connections 2024-01-21T19:55:39.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.607+0000 7f8efa3ed700 1 -- 172.21.15.173:0/3484953343 wait complete. 2024-01-21T19:55:39.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.608+0000 7f8efa3ed700 1 Processor -- start 2024-01-21T19:55:39.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.608+0000 7f8efa3ed700 1 -- start start 2024-01-21T19:55:39.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.608+0000 7f8efa3ed700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ef40f5410 0x7f8ef41093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:39.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.608+0000 7f8efa3ed700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef4060280 con 0x7f8ef40f5410 2024-01-21T19:55:39.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.608+0000 7f8ef3fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ef40f5410 0x7f8ef41093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:39.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.608+0000 7f8ef3fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ef40f5410 0x7f8ef41093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:38172/0 (socket says 172.21.15.173:38172) 2024-01-21T19:55:39.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.608+0000 7f8ef3fff700 1 -- 172.21.15.173:0/992378140 learned_addr learned my addr 172.21.15.173:0/992378140 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:39.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.609+0000 7f8ef3fff700 1 -- 172.21.15.173:0/992378140 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee4005040 con 0x7f8ef40f5410 2024-01-21T19:55:39.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.609+0000 7f8ef3fff700 1 --2- 172.21.15.173:0/992378140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ef40f5410 0x7f8ef41093f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f8ee4013510 tx=0x7f8ee400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:39.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.609+0000 7f8ef0ff9700 1 -- 172.21.15.173:0/992378140 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ee4009320 con 0x7f8ef40f5410 2024-01-21T19:55:39.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.609+0000 7f8efa3ed700 1 -- 172.21.15.173:0/992378140 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef4060480 con 0x7f8ef40f5410 2024-01-21T19:55:39.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.609+0000 7f8ef0ff9700 1 -- 172.21.15.173:0/992378140 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ee4013f00 con 0x7f8ef40f5410 2024-01-21T19:55:39.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.609+0000 7f8ef0ff9700 1 -- 172.21.15.173:0/992378140 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ee4014d90 con 0x7f8ef40f5410 2024-01-21T19:55:39.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.609+0000 7f8efa3ed700 1 -- 172.21.15.173:0/992378140 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef4109b20 con 0x7f8ef40f5410 2024-01-21T19:55:39.602 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.610+0000 7f8ef0ff9700 1 -- 172.21.15.173:0/992378140 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ee40095f0 con 0x7f8ef40f5410 2024-01-21T19:55:39.602 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.610+0000 7f8efa3ed700 1 -- 172.21.15.173:0/992378140 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ef4045ad0 con 0x7f8ef40f5410 2024-01-21T19:55:39.602 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.610+0000 7f8ef0ff9700 1 --2- 172.21.15.173:0/992378140 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8edc02ff60 0x7f8edc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:39.602 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.610+0000 7f8ef0ff9700 1 -- 172.21.15.173:0/992378140 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ee4040fe0 con 0x7f8ef40f5410 2024-01-21T19:55:39.602 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.610+0000 7f8ef37fe700 1 --2- 172.21.15.173:0/992378140 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8edc02ff60 0x7f8edc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:39.602 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.611+0000 7f8ef37fe700 1 --2- 172.21.15.173:0/992378140 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8edc02ff60 0x7f8edc032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f8ee80096a0 tx=0x7f8ee8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:39.606 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.615+0000 7f8ef0ff9700 1 -- 172.21.15.173:0/992378140 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ee401d7f0 con 0x7f8ef40f5410 2024-01-21T19:55:39.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.821+0000 7f8efa3ed700 1 -- 172.21.15.173:0/992378140 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ef410a340 con 0x7f8ef40f5410 2024-01-21T19:55:39.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.822+0000 7f8ef0ff9700 1 -- 172.21.15.173:0/992378140 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8ee4010230 con 0x7f8ef40f5410 2024-01-21T19:55:39.813 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:39.814 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:39.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.823+0000 7f8efa3ed700 1 -- 172.21.15.173:0/992378140 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8edc02ff60 msgr2=0x7f8edc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:39.816 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.823+0000 7f8efa3ed700 1 --2- 172.21.15.173:0/992378140 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8edc02ff60 0x7f8edc032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f8ee80096a0 tx=0x7f8ee8006b40).stop 2024-01-21T19:55:39.816 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.823+0000 7f8efa3ed700 1 -- 172.21.15.173:0/992378140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ef40f5410 msgr2=0x7f8ef41093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:39.816 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.823+0000 7f8efa3ed700 1 --2- 172.21.15.173:0/992378140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ef40f5410 0x7f8ef41093f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f8ee4013510 tx=0x7f8ee400b040).stop 2024-01-21T19:55:39.816 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.824+0000 7f8efa3ed700 1 -- 172.21.15.173:0/992378140 shutdown_connections 2024-01-21T19:55:39.816 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.824+0000 7f8efa3ed700 1 --2- 172.21.15.173:0/992378140 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8edc02ff60 0x7f8edc032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:39.816 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.824+0000 7f8efa3ed700 1 --2- 172.21.15.173:0/992378140 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ef40f5410 0x7f8ef41093f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:39.816 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.824+0000 7f8efa3ed700 1 -- 172.21.15.173:0/992378140 >> 172.21.15.173:0/992378140 conn(0x7f8ef40f0b70 msgr2=0x7f8ef40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:39.816 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.824+0000 7f8efa3ed700 1 -- 172.21.15.173:0/992378140 shutdown_connections 2024-01-21T19:55:39.816 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:39.824+0000 7f8efa3ed700 1 -- 172.21.15.173:0/992378140 wait complete. 2024-01-21T19:55:39.816 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:55:40.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:39 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:39.822754+0000 mon.smithi169 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.173:0/992378140' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:55:41.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:40 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:39.001444+0000 mgr.smithi169.espgue (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:41.240 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:55:41.240 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:42.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.440+0000 7f345e9d9700 1 -- 172.21.15.173:0/81785306 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34580f6510 msgr2=0x7f34580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:42.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.440+0000 7f345e9d9700 1 --2- 172.21.15.173:0/81785306 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34580f6510 0x7f34580f6930 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f3448009fb0 tx=0x7f3448009b30).stop 2024-01-21T19:55:42.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.440+0000 7f345e9d9700 1 -- 172.21.15.173:0/81785306 shutdown_connections 2024-01-21T19:55:42.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.440+0000 7f345e9d9700 1 --2- 172.21.15.173:0/81785306 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34580f6510 0x7f34580f6930 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:42.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.440+0000 7f345e9d9700 1 -- 172.21.15.173:0/81785306 >> 172.21.15.173:0/81785306 conn(0x7f34580f1c70 msgr2=0x7f34580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:42.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.440+0000 7f345e9d9700 1 -- 172.21.15.173:0/81785306 shutdown_connections 2024-01-21T19:55:42.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.440+0000 7f345e9d9700 1 -- 172.21.15.173:0/81785306 wait complete. 2024-01-21T19:55:42.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.441+0000 7f345e9d9700 1 Processor -- start 2024-01-21T19:55:42.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.441+0000 7f345e9d9700 1 -- start start 2024-01-21T19:55:42.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.441+0000 7f345e9d9700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34580f6510 0x7f3458106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:42.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.441+0000 7f345e9d9700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3458106c20 con 0x7f34580f6510 2024-01-21T19:55:42.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.441+0000 7f3457fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34580f6510 0x7f3458106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:42.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.442+0000 7f3457fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34580f6510 0x7f3458106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:38198/0 (socket says 172.21.15.173:38198) 2024-01-21T19:55:42.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.442+0000 7f3457fff700 1 -- 172.21.15.173:0/1434691298 learned_addr learned my addr 172.21.15.173:0/1434691298 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:42.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.442+0000 7f3457fff700 1 -- 172.21.15.173:0/1434691298 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3448005040 con 0x7f34580f6510 2024-01-21T19:55:42.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.442+0000 7f3457fff700 1 --2- 172.21.15.173:0/1434691298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34580f6510 0x7f3458106730 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f3448013510 tx=0x7f344800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:42.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.442+0000 7f3454ff9700 1 -- 172.21.15.173:0/1434691298 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3448009320 con 0x7f34580f6510 2024-01-21T19:55:42.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.442+0000 7f3454ff9700 1 -- 172.21.15.173:0/1434691298 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3448013f00 con 0x7f34580f6510 2024-01-21T19:55:42.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.442+0000 7f345e9d9700 1 -- 172.21.15.173:0/1434691298 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3458106e20 con 0x7f34580f6510 2024-01-21T19:55:42.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.442+0000 7f3454ff9700 1 -- 172.21.15.173:0/1434691298 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3448014d90 con 0x7f34580f6510 2024-01-21T19:55:42.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.442+0000 7f345e9d9700 1 -- 172.21.15.173:0/1434691298 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34581072b0 con 0x7f34580f6510 2024-01-21T19:55:42.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.443+0000 7f3454ff9700 1 -- 172.21.15.173:0/1434691298 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34480095f0 con 0x7f34580f6510 2024-01-21T19:55:42.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.443+0000 7f345e9d9700 1 -- 172.21.15.173:0/1434691298 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3458045ad0 con 0x7f34580f6510 2024-01-21T19:55:42.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.443+0000 7f3454ff9700 1 --2- 172.21.15.173:0/1434691298 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f344002ff60 0x7f3440032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:42.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.443+0000 7f3454ff9700 1 -- 172.21.15.173:0/1434691298 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f34480410c0 con 0x7f34580f6510 2024-01-21T19:55:42.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.443+0000 7f34577fe700 1 --2- 172.21.15.173:0/1434691298 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f344002ff60 0x7f3440032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:42.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.444+0000 7f34577fe700 1 --2- 172.21.15.173:0/1434691298 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f344002ff60 0x7f3440032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f344c0096a0 tx=0x7f344c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:42.439 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.448+0000 7f3454ff9700 1 -- 172.21.15.173:0/1434691298 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f344801d7f0 con 0x7f34580f6510 2024-01-21T19:55:42.644 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.653+0000 7f345e9d9700 1 -- 172.21.15.173:0/1434691298 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34580fbf60 con 0x7f34580f6510 2024-01-21T19:55:42.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.654+0000 7f3454ff9700 1 -- 172.21.15.173:0/1434691298 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3448010230 con 0x7f34580f6510 2024-01-21T19:55:42.645 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:42.645 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:42.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.655+0000 7f345e9d9700 1 -- 172.21.15.173:0/1434691298 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f344002ff60 msgr2=0x7f3440032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:42.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.655+0000 7f345e9d9700 1 --2- 172.21.15.173:0/1434691298 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f344002ff60 0x7f3440032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f344c0096a0 tx=0x7f344c006b40).stop 2024-01-21T19:55:42.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.655+0000 7f345e9d9700 1 -- 172.21.15.173:0/1434691298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34580f6510 msgr2=0x7f3458106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:42.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.655+0000 7f345e9d9700 1 --2- 172.21.15.173:0/1434691298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34580f6510 0x7f3458106730 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f3448013510 tx=0x7f344800b040).stop 2024-01-21T19:55:42.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.655+0000 7f345e9d9700 1 -- 172.21.15.173:0/1434691298 shutdown_connections 2024-01-21T19:55:42.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.655+0000 7f345e9d9700 1 --2- 172.21.15.173:0/1434691298 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f344002ff60 0x7f3440032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:42.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.655+0000 7f345e9d9700 1 --2- 172.21.15.173:0/1434691298 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f34580f6510 0x7f3458106730 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:42.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.655+0000 7f345e9d9700 1 -- 172.21.15.173:0/1434691298 >> 172.21.15.173:0/1434691298 conn(0x7f34580f1c70 msgr2=0x7f34580f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:42.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.655+0000 7f345e9d9700 1 -- 172.21.15.173:0/1434691298 shutdown_connections 2024-01-21T19:55:42.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:42.655+0000 7f345e9d9700 1 -- 172.21.15.173:0/1434691298 wait complete. 2024-01-21T19:55:42.648 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:55:43.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:41.001705+0000 mgr.smithi169.espgue (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:43.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:42.654402+0000 mon.smithi169 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.173:0/1434691298' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:55:44.039 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:55:44.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:45.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:44 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:43.002006+0000 mgr.smithi169.espgue (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:45.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.206+0000 7fc5c1717700 1 -- 172.21.15.173:0/1051621872 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5bc060140 msgr2=0x7fc5bc0f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:45.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.206+0000 7fc5c1717700 1 --2- 172.21.15.173:0/1051621872 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5bc060140 0x7fc5bc0f8fc0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fc5ac009fb0 tx=0x7fc5ac009b30).stop 2024-01-21T19:55:45.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.206+0000 7fc5c1717700 1 -- 172.21.15.173:0/1051621872 shutdown_connections 2024-01-21T19:55:45.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.206+0000 7fc5c1717700 1 --2- 172.21.15.173:0/1051621872 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5bc060140 0x7fc5bc0f8fc0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:45.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.206+0000 7fc5c1717700 1 -- 172.21.15.173:0/1051621872 >> 172.21.15.173:0/1051621872 conn(0x7fc5bc0f09a0 msgr2=0x7fc5bc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:45.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.206+0000 7fc5c1717700 1 -- 172.21.15.173:0/1051621872 shutdown_connections 2024-01-21T19:55:45.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.206+0000 7fc5c1717700 1 -- 172.21.15.173:0/1051621872 wait complete. 2024-01-21T19:55:45.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.206+0000 7fc5c1717700 1 Processor -- start 2024-01-21T19:55:45.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.206+0000 7fc5c1717700 1 -- start start 2024-01-21T19:55:45.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.207+0000 7fc5c1717700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5bc060140 0x7fc5bc106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:45.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.207+0000 7fc5c1717700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc106a10 con 0x7fc5bc060140 2024-01-21T19:55:45.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.207+0000 7fc5bbfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5bc060140 0x7fc5bc106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:45.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.207+0000 7fc5bbfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5bc060140 0x7fc5bc106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:41726/0 (socket says 172.21.15.173:41726) 2024-01-21T19:55:45.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.207+0000 7fc5bbfff700 1 -- 172.21.15.173:0/3954439700 learned_addr learned my addr 172.21.15.173:0/3954439700 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:45.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.207+0000 7fc5bbfff700 1 -- 172.21.15.173:0/3954439700 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ac005040 con 0x7fc5bc060140 2024-01-21T19:55:45.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.207+0000 7fc5bbfff700 1 --2- 172.21.15.173:0/3954439700 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5bc060140 0x7fc5bc106520 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fc5ac013510 tx=0x7fc5ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:45.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.208+0000 7fc5b8ff9700 1 -- 172.21.15.173:0/3954439700 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5ac009320 con 0x7fc5bc060140 2024-01-21T19:55:45.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.208+0000 7fc5c1717700 1 -- 172.21.15.173:0/3954439700 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5bc106c10 con 0x7fc5bc060140 2024-01-21T19:55:45.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.208+0000 7fc5c1717700 1 -- 172.21.15.173:0/3954439700 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5bc1070a0 con 0x7fc5bc060140 2024-01-21T19:55:45.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.208+0000 7fc5b8ff9700 1 -- 172.21.15.173:0/3954439700 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5ac013f00 con 0x7fc5bc060140 2024-01-21T19:55:45.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.208+0000 7fc5b8ff9700 1 -- 172.21.15.173:0/3954439700 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5ac014d60 con 0x7fc5bc060140 2024-01-21T19:55:45.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.208+0000 7fc5b8ff9700 1 -- 172.21.15.173:0/3954439700 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc5ac014420 con 0x7fc5bc060140 2024-01-21T19:55:45.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.208+0000 7fc5c1717700 1 -- 172.21.15.173:0/3954439700 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc59c004fa0 con 0x7fc5bc060140 2024-01-21T19:55:45.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.209+0000 7fc5b8ff9700 1 --2- 172.21.15.173:0/3954439700 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc5a402ff60 0x7fc5a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:45.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.209+0000 7fc5b8ff9700 1 -- 172.21.15.173:0/3954439700 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc5ac0412e0 con 0x7fc5bc060140 2024-01-21T19:55:45.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.209+0000 7fc5bb7fe700 1 --2- 172.21.15.173:0/3954439700 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc5a402ff60 0x7fc5a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:45.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.209+0000 7fc5bb7fe700 1 --2- 172.21.15.173:0/3954439700 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc5a402ff60 0x7fc5a4032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fc5b00096a0 tx=0x7fc5b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:45.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.213+0000 7fc5b8ff9700 1 -- 172.21.15.173:0/3954439700 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc5ac012b20 con 0x7fc5bc060140 2024-01-21T19:55:45.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.420+0000 7fc5c1717700 1 -- 172.21.15.173:0/3954439700 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc59c005670 con 0x7fc5bc060140 2024-01-21T19:55:45.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.421+0000 7fc5b8ff9700 1 -- 172.21.15.173:0/3954439700 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc5ac012b20 con 0x7fc5bc060140 2024-01-21T19:55:45.413 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:45.413 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:45.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.422+0000 7fc5c1717700 1 -- 172.21.15.173:0/3954439700 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc5a402ff60 msgr2=0x7fc5a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:45.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.422+0000 7fc5c1717700 1 --2- 172.21.15.173:0/3954439700 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc5a402ff60 0x7fc5a4032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fc5b00096a0 tx=0x7fc5b0006b40).stop 2024-01-21T19:55:45.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.422+0000 7fc5c1717700 1 -- 172.21.15.173:0/3954439700 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5bc060140 msgr2=0x7fc5bc106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:45.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.422+0000 7fc5c1717700 1 --2- 172.21.15.173:0/3954439700 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5bc060140 0x7fc5bc106520 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fc5ac013510 tx=0x7fc5ac00b040).stop 2024-01-21T19:55:45.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.422+0000 7fc5c1717700 1 -- 172.21.15.173:0/3954439700 shutdown_connections 2024-01-21T19:55:45.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.422+0000 7fc5c1717700 1 --2- 172.21.15.173:0/3954439700 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc5a402ff60 0x7fc5a4032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:45.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.422+0000 7fc5c1717700 1 --2- 172.21.15.173:0/3954439700 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5bc060140 0x7fc5bc106520 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:45.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.422+0000 7fc5c1717700 1 -- 172.21.15.173:0/3954439700 >> 172.21.15.173:0/3954439700 conn(0x7fc5bc0f09a0 msgr2=0x7fc5bc0f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:45.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.422+0000 7fc5c1717700 1 -- 172.21.15.173:0/3954439700 shutdown_connections 2024-01-21T19:55:45.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:45.423+0000 7fc5c1717700 1 -- 172.21.15.173:0/3954439700 wait complete. 2024-01-21T19:55:45.416 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:55:46.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:45 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:45.422440+0000 mon.smithi169 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.173:0/3954439700' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:55:46.802 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:55:46.802 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:47.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:46 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:45.002361+0000 mgr.smithi169.espgue (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:47.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.974+0000 7f6e30337700 1 -- 172.21.15.173:0/180812388 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6e280f5410 msgr2=0x7f6e280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:47.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.974+0000 7f6e30337700 1 --2- 172.21.15.173:0/180812388 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6e280f5410 0x7f6e280f5830 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f6e18009fb0 tx=0x7f6e18009b30).stop 2024-01-21T19:55:47.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.974+0000 7f6e30337700 1 -- 172.21.15.173:0/180812388 shutdown_connections 2024-01-21T19:55:47.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.974+0000 7f6e30337700 1 --2- 172.21.15.173:0/180812388 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6e280f5410 0x7f6e280f5830 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:47.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.974+0000 7f6e30337700 1 -- 172.21.15.173:0/180812388 >> 172.21.15.173:0/180812388 conn(0x7f6e280f0b70 msgr2=0x7f6e280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:47.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.974+0000 7f6e30337700 1 -- 172.21.15.173:0/180812388 shutdown_connections 2024-01-21T19:55:47.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.974+0000 7f6e30337700 1 -- 172.21.15.173:0/180812388 wait complete. 2024-01-21T19:55:47.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.974+0000 7f6e30337700 1 Processor -- start 2024-01-21T19:55:47.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.975+0000 7f6e30337700 1 -- start start 2024-01-21T19:55:47.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.975+0000 7f6e30337700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6e280f5410 0x7f6e28109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:47.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.975+0000 7f6e30337700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e28060240 con 0x7f6e280f5410 2024-01-21T19:55:47.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.975+0000 7f6e2e0d3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6e280f5410 0x7f6e28109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:47.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.975+0000 7f6e2e0d3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6e280f5410 0x7f6e28109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:41732/0 (socket says 172.21.15.173:41732) 2024-01-21T19:55:47.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.975+0000 7f6e2e0d3700 1 -- 172.21.15.173:0/2486703241 learned_addr learned my addr 172.21.15.173:0/2486703241 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:47.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.975+0000 7f6e2e0d3700 1 -- 172.21.15.173:0/2486703241 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e18005040 con 0x7f6e280f5410 2024-01-21T19:55:47.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.975+0000 7f6e2e0d3700 1 --2- 172.21.15.173:0/2486703241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6e280f5410 0x7f6e28109430 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f6e18013510 tx=0x7f6e1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:47.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.975+0000 7f6e1effd700 1 -- 172.21.15.173:0/2486703241 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e18009320 con 0x7f6e280f5410 2024-01-21T19:55:47.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.975+0000 7f6e30337700 1 -- 172.21.15.173:0/2486703241 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e28060440 con 0x7f6e280f5410 2024-01-21T19:55:47.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.975+0000 7f6e1effd700 1 -- 172.21.15.173:0/2486703241 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e18013f00 con 0x7f6e280f5410 2024-01-21T19:55:47.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.976+0000 7f6e1effd700 1 -- 172.21.15.173:0/2486703241 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e18014d90 con 0x7f6e280f5410 2024-01-21T19:55:47.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.976+0000 7f6e30337700 1 -- 172.21.15.173:0/2486703241 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e28109ae0 con 0x7f6e280f5410 2024-01-21T19:55:47.970 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.976+0000 7f6e1effd700 1 -- 172.21.15.173:0/2486703241 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6e180095f0 con 0x7f6e280f5410 2024-01-21T19:55:47.970 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.977+0000 7f6e30337700 1 -- 172.21.15.173:0/2486703241 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e28045ad0 con 0x7f6e280f5410 2024-01-21T19:55:47.971 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.977+0000 7f6e1effd700 1 --2- 172.21.15.173:0/2486703241 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f6e1402ff60 0x7f6e14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:47.971 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.977+0000 7f6e1effd700 1 -- 172.21.15.173:0/2486703241 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6e18041000 con 0x7f6e280f5410 2024-01-21T19:55:47.971 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.977+0000 7f6e2d8d2700 1 --2- 172.21.15.173:0/2486703241 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f6e1402ff60 0x7f6e14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:47.971 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.977+0000 7f6e2d8d2700 1 --2- 172.21.15.173:0/2486703241 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f6e1402ff60 0x7f6e14032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f6e240096a0 tx=0x7f6e24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:47.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:47.982+0000 7f6e1effd700 1 -- 172.21.15.173:0/2486703241 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6e18014420 con 0x7f6e280f5410 2024-01-21T19:55:48.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:48.187+0000 7f6e30337700 1 -- 172.21.15.173:0/2486703241 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e280fae60 con 0x7f6e280f5410 2024-01-21T19:55:48.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:48.187+0000 7f6e1effd700 1 -- 172.21.15.173:0/2486703241 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6e1801d7f0 con 0x7f6e280f5410 2024-01-21T19:55:48.180 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:48.180 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:48.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:48.189+0000 7f6e30337700 1 -- 172.21.15.173:0/2486703241 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f6e1402ff60 msgr2=0x7f6e14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:48.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:48.189+0000 7f6e30337700 1 --2- 172.21.15.173:0/2486703241 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f6e1402ff60 0x7f6e14032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f6e240096a0 tx=0x7f6e24006b40).stop 2024-01-21T19:55:48.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:48.189+0000 7f6e30337700 1 -- 172.21.15.173:0/2486703241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6e280f5410 msgr2=0x7f6e28109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:48.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:48.189+0000 7f6e30337700 1 --2- 172.21.15.173:0/2486703241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6e280f5410 0x7f6e28109430 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f6e18013510 tx=0x7f6e1800b040).stop 2024-01-21T19:55:48.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:48.189+0000 7f6e30337700 1 -- 172.21.15.173:0/2486703241 shutdown_connections 2024-01-21T19:55:48.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:48.189+0000 7f6e30337700 1 --2- 172.21.15.173:0/2486703241 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f6e1402ff60 0x7f6e14032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:48.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:48.189+0000 7f6e30337700 1 --2- 172.21.15.173:0/2486703241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6e280f5410 0x7f6e28109430 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:48.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:48.189+0000 7f6e30337700 1 -- 172.21.15.173:0/2486703241 >> 172.21.15.173:0/2486703241 conn(0x7f6e280f0b70 msgr2=0x7f6e280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:48.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:48.189+0000 7f6e30337700 1 -- 172.21.15.173:0/2486703241 shutdown_connections 2024-01-21T19:55:48.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:48.189+0000 7f6e30337700 1 -- 172.21.15.173:0/2486703241 wait complete. 2024-01-21T19:55:48.183 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:55:49.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:47.002777+0000 mgr.smithi169.espgue (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:49.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:48.189313+0000 mon.smithi169 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.173:0/2486703241' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:55:49.584 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:55:49.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:50.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.771+0000 7fc5d7ae8700 1 -- 172.21.15.173:0/2885764893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5d00f6510 msgr2=0x7fc5d00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:50.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.771+0000 7fc5d7ae8700 1 --2- 172.21.15.173:0/2885764893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5d00f6510 0x7fc5d00f6930 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fc5c0009fb0 tx=0x7fc5c0009b30).stop 2024-01-21T19:55:50.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.771+0000 7fc5d7ae8700 1 -- 172.21.15.173:0/2885764893 shutdown_connections 2024-01-21T19:55:50.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.771+0000 7fc5d7ae8700 1 --2- 172.21.15.173:0/2885764893 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5d00f6510 0x7fc5d00f6930 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:50.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.771+0000 7fc5d7ae8700 1 -- 172.21.15.173:0/2885764893 >> 172.21.15.173:0/2885764893 conn(0x7fc5d00f1c70 msgr2=0x7fc5d00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:50.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.771+0000 7fc5d7ae8700 1 -- 172.21.15.173:0/2885764893 shutdown_connections 2024-01-21T19:55:50.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.771+0000 7fc5d7ae8700 1 -- 172.21.15.173:0/2885764893 wait complete. 2024-01-21T19:55:50.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.772+0000 7fc5d7ae8700 1 Processor -- start 2024-01-21T19:55:50.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.772+0000 7fc5d7ae8700 1 -- start start 2024-01-21T19:55:50.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.772+0000 7fc5d7ae8700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5d00f6510 0x7fc5d0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:50.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.772+0000 7fc5d7ae8700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d0106c20 con 0x7fc5d00f6510 2024-01-21T19:55:50.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.773+0000 7fc5d5884700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5d00f6510 0x7fc5d0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:50.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.773+0000 7fc5d5884700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5d00f6510 0x7fc5d0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:41750/0 (socket says 172.21.15.173:41750) 2024-01-21T19:55:50.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.773+0000 7fc5d5884700 1 -- 172.21.15.173:0/3050072644 learned_addr learned my addr 172.21.15.173:0/3050072644 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:50.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.773+0000 7fc5d5884700 1 -- 172.21.15.173:0/3050072644 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c0005040 con 0x7fc5d00f6510 2024-01-21T19:55:50.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.773+0000 7fc5d5884700 1 --2- 172.21.15.173:0/3050072644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5d00f6510 0x7fc5d0106730 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fc5c0013510 tx=0x7fc5c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:50.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.773+0000 7fc5c67fc700 1 -- 172.21.15.173:0/3050072644 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5c0009320 con 0x7fc5d00f6510 2024-01-21T19:55:50.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.773+0000 7fc5d7ae8700 1 -- 172.21.15.173:0/3050072644 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d0106e20 con 0x7fc5d00f6510 2024-01-21T19:55:50.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.773+0000 7fc5c67fc700 1 -- 172.21.15.173:0/3050072644 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5c0013f00 con 0x7fc5d00f6510 2024-01-21T19:55:50.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.773+0000 7fc5c67fc700 1 -- 172.21.15.173:0/3050072644 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5c0014d90 con 0x7fc5d00f6510 2024-01-21T19:55:50.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.773+0000 7fc5d7ae8700 1 -- 172.21.15.173:0/3050072644 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d01072b0 con 0x7fc5d00f6510 2024-01-21T19:55:50.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.774+0000 7fc5c67fc700 1 -- 172.21.15.173:0/3050072644 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc5c00095f0 con 0x7fc5d00f6510 2024-01-21T19:55:50.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.774+0000 7fc5d7ae8700 1 -- 172.21.15.173:0/3050072644 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d0045ad0 con 0x7fc5d00f6510 2024-01-21T19:55:50.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.774+0000 7fc5c67fc700 1 --2- 172.21.15.173:0/3050072644 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc5bc02ffb0 0x7fc5bc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:50.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.774+0000 7fc5c67fc700 1 -- 172.21.15.173:0/3050072644 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc5c00410c0 con 0x7fc5d00f6510 2024-01-21T19:55:50.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.775+0000 7fc5d5083700 1 --2- 172.21.15.173:0/3050072644 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc5bc02ffb0 0x7fc5bc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:50.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.775+0000 7fc5d5083700 1 --2- 172.21.15.173:0/3050072644 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc5bc02ffb0 0x7fc5bc032470 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc5cc0096a0 tx=0x7fc5cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:50.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.779+0000 7fc5c67fc700 1 -- 172.21.15.173:0/3050072644 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc5c001d7f0 con 0x7fc5d00f6510 2024-01-21T19:55:50.976 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.984+0000 7fc5d7ae8700 1 -- 172.21.15.173:0/3050072644 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc5d00fbf60 con 0x7fc5d00f6510 2024-01-21T19:55:50.977 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.984+0000 7fc5c67fc700 1 -- 172.21.15.173:0/3050072644 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc5c0010230 con 0x7fc5d00f6510 2024-01-21T19:55:50.977 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:50.977 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:50.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.986+0000 7fc5d7ae8700 1 -- 172.21.15.173:0/3050072644 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc5bc02ffb0 msgr2=0x7fc5bc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:50.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.986+0000 7fc5d7ae8700 1 --2- 172.21.15.173:0/3050072644 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc5bc02ffb0 0x7fc5bc032470 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc5cc0096a0 tx=0x7fc5cc006b40).stop 2024-01-21T19:55:50.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.986+0000 7fc5d7ae8700 1 -- 172.21.15.173:0/3050072644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5d00f6510 msgr2=0x7fc5d0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:50.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.986+0000 7fc5d7ae8700 1 --2- 172.21.15.173:0/3050072644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5d00f6510 0x7fc5d0106730 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fc5c0013510 tx=0x7fc5c000b040).stop 2024-01-21T19:55:50.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.986+0000 7fc5d7ae8700 1 -- 172.21.15.173:0/3050072644 shutdown_connections 2024-01-21T19:55:50.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.986+0000 7fc5d7ae8700 1 --2- 172.21.15.173:0/3050072644 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc5bc02ffb0 0x7fc5bc032470 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:50.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.986+0000 7fc5d7ae8700 1 --2- 172.21.15.173:0/3050072644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc5d00f6510 0x7fc5d0106730 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:50.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.986+0000 7fc5d7ae8700 1 -- 172.21.15.173:0/3050072644 >> 172.21.15.173:0/3050072644 conn(0x7fc5d00f1c70 msgr2=0x7fc5d00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:50.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.986+0000 7fc5d7ae8700 1 -- 172.21.15.173:0/3050072644 shutdown_connections 2024-01-21T19:55:50.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:50.986+0000 7fc5d7ae8700 1 -- 172.21.15.173:0/3050072644 wait complete. 2024-01-21T19:55:50.980 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:55:51.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:50 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:49.003184+0000 mgr.smithi169.espgue (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:52.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:51 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:50.986219+0000 mon.smithi169 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.173:0/3050072644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:55:52.370 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:55:52.370 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:53.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:52 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:51.003536+0000 mgr.smithi169.espgue (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:53.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.599+0000 7fc482a7b700 1 -- 172.21.15.173:0/156928243 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc47c0f5410 msgr2=0x7fc47c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:53.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.599+0000 7fc482a7b700 1 --2- 172.21.15.173:0/156928243 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc47c0f5410 0x7fc47c0f5830 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fc474009960 tx=0x7fc474004d10).stop 2024-01-21T19:55:53.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.600+0000 7fc482a7b700 1 -- 172.21.15.173:0/156928243 shutdown_connections 2024-01-21T19:55:53.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.600+0000 7fc482a7b700 1 --2- 172.21.15.173:0/156928243 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc47c0f5410 0x7fc47c0f5830 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:53.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.600+0000 7fc482a7b700 1 -- 172.21.15.173:0/156928243 >> 172.21.15.173:0/156928243 conn(0x7fc47c0f0b70 msgr2=0x7fc47c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:53.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.600+0000 7fc482a7b700 1 -- 172.21.15.173:0/156928243 shutdown_connections 2024-01-21T19:55:53.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.600+0000 7fc482a7b700 1 -- 172.21.15.173:0/156928243 wait complete. 2024-01-21T19:55:53.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.600+0000 7fc482a7b700 1 Processor -- start 2024-01-21T19:55:53.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.600+0000 7fc482a7b700 1 -- start start 2024-01-21T19:55:53.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.600+0000 7fc482a7b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc47c0f5410 0x7fc47c1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:53.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.600+0000 7fc482a7b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c060280 con 0x7fc47c0f5410 2024-01-21T19:55:53.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.601+0000 7fc480817700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc47c0f5410 0x7fc47c1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:53.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.601+0000 7fc480817700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc47c0f5410 0x7fc47c1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:37436/0 (socket says 172.21.15.173:37436) 2024-01-21T19:55:53.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.601+0000 7fc480817700 1 -- 172.21.15.173:0/1697899957 learned_addr learned my addr 172.21.15.173:0/1697899957 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:53.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.601+0000 7fc480817700 1 -- 172.21.15.173:0/1697899957 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc474009640 con 0x7fc47c0f5410 2024-01-21T19:55:53.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.601+0000 7fc480817700 1 --2- 172.21.15.173:0/1697899957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc47c0f5410 0x7fc47c1093f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fc474004b70 tx=0x7fc474004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:53.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.601+0000 7fc4797fa700 1 -- 172.21.15.173:0/1697899957 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc474018070 con 0x7fc47c0f5410 2024-01-21T19:55:53.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.601+0000 7fc482a7b700 1 -- 172.21.15.173:0/1697899957 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc47c060480 con 0x7fc47c0f5410 2024-01-21T19:55:53.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.601+0000 7fc4797fa700 1 -- 172.21.15.173:0/1697899957 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc47400b870 con 0x7fc47c0f5410 2024-01-21T19:55:53.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.601+0000 7fc4797fa700 1 -- 172.21.15.173:0/1697899957 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc47401c560 con 0x7fc47c0f5410 2024-01-21T19:55:53.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.601+0000 7fc482a7b700 1 -- 172.21.15.173:0/1697899957 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc47c109b20 con 0x7fc47c0f5410 2024-01-21T19:55:53.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.602+0000 7fc4797fa700 1 -- 172.21.15.173:0/1697899957 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc474013630 con 0x7fc47c0f5410 2024-01-21T19:55:53.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.602+0000 7fc482a7b700 1 -- 172.21.15.173:0/1697899957 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc47c045ad0 con 0x7fc47c0f5410 2024-01-21T19:55:53.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.602+0000 7fc4797fa700 1 --2- 172.21.15.173:0/1697899957 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc46802ffb0 0x7fc468032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:53.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.602+0000 7fc4797fa700 1 -- 172.21.15.173:0/1697899957 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc474041290 con 0x7fc47c0f5410 2024-01-21T19:55:53.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.602+0000 7fc47bfff700 1 --2- 172.21.15.173:0/1697899957 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc46802ffb0 0x7fc468032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:53.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.603+0000 7fc47bfff700 1 --2- 172.21.15.173:0/1697899957 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc46802ffb0 0x7fc468032470 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fc46c0096a0 tx=0x7fc46c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:53.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.607+0000 7fc4797fa700 1 -- 172.21.15.173:0/1697899957 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc474004810 con 0x7fc47c0f5410 2024-01-21T19:55:53.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.812+0000 7fc482a7b700 1 -- 172.21.15.173:0/1697899957 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc47c10a340 con 0x7fc47c0f5410 2024-01-21T19:55:53.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.813+0000 7fc4797fa700 1 -- 172.21.15.173:0/1697899957 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc474012380 con 0x7fc47c0f5410 2024-01-21T19:55:53.806 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:53.806 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:53.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.815+0000 7fc482a7b700 1 -- 172.21.15.173:0/1697899957 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc46802ffb0 msgr2=0x7fc468032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:53.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.815+0000 7fc482a7b700 1 --2- 172.21.15.173:0/1697899957 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc46802ffb0 0x7fc468032470 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fc46c0096a0 tx=0x7fc46c006b40).stop 2024-01-21T19:55:53.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.815+0000 7fc482a7b700 1 -- 172.21.15.173:0/1697899957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc47c0f5410 msgr2=0x7fc47c1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:53.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.815+0000 7fc482a7b700 1 --2- 172.21.15.173:0/1697899957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc47c0f5410 0x7fc47c1093f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fc474004b70 tx=0x7fc474004d10).stop 2024-01-21T19:55:53.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.815+0000 7fc482a7b700 1 -- 172.21.15.173:0/1697899957 shutdown_connections 2024-01-21T19:55:53.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.815+0000 7fc482a7b700 1 --2- 172.21.15.173:0/1697899957 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fc46802ffb0 0x7fc468032470 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:53.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.815+0000 7fc482a7b700 1 --2- 172.21.15.173:0/1697899957 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc47c0f5410 0x7fc47c1093f0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:53.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.815+0000 7fc482a7b700 1 -- 172.21.15.173:0/1697899957 >> 172.21.15.173:0/1697899957 conn(0x7fc47c0f0b70 msgr2=0x7fc47c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:53.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.815+0000 7fc482a7b700 1 -- 172.21.15.173:0/1697899957 shutdown_connections 2024-01-21T19:55:53.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:53.815+0000 7fc482a7b700 1 -- 172.21.15.173:0/1697899957 wait complete. 2024-01-21T19:55:53.808 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:55:54.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:53 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:53.814998+0000 mon.smithi169 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.173:0/1697899957' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:55:55.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:54 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:53.003803+0000 mgr.smithi169.espgue (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:55.202 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:55:55.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:56.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.451+0000 7fe659f1a700 1 -- 172.21.15.173:0/256302530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe654060280 msgr2=0x7fe6540f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:56.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.451+0000 7fe659f1a700 1 --2- 172.21.15.173:0/256302530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe654060280 0x7fe6540f90f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fe63c009fb0 tx=0x7fe63c009b30).stop 2024-01-21T19:55:56.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.451+0000 7fe659f1a700 1 -- 172.21.15.173:0/256302530 shutdown_connections 2024-01-21T19:55:56.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.451+0000 7fe659f1a700 1 --2- 172.21.15.173:0/256302530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe654060280 0x7fe6540f90f0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:56.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.451+0000 7fe659f1a700 1 -- 172.21.15.173:0/256302530 >> 172.21.15.173:0/256302530 conn(0x7fe6540f0b70 msgr2=0x7fe6540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:56.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.451+0000 7fe659f1a700 1 -- 172.21.15.173:0/256302530 shutdown_connections 2024-01-21T19:55:56.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.451+0000 7fe659f1a700 1 -- 172.21.15.173:0/256302530 wait complete. 2024-01-21T19:55:56.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.451+0000 7fe659f1a700 1 Processor -- start 2024-01-21T19:55:56.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.452+0000 7fe659f1a700 1 -- start start 2024-01-21T19:55:56.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.452+0000 7fe659f1a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe654060280 0x7fe6541088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:56.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.452+0000 7fe659f1a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe654108de0 con 0x7fe654060280 2024-01-21T19:55:56.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.452+0000 7fe6537fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe654060280 0x7fe6541088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:56.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.452+0000 7fe6537fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe654060280 0x7fe6541088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:37466/0 (socket says 172.21.15.173:37466) 2024-01-21T19:55:56.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.452+0000 7fe6537fe700 1 -- 172.21.15.173:0/4208218136 learned_addr learned my addr 172.21.15.173:0/4208218136 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:56.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.452+0000 7fe6537fe700 1 -- 172.21.15.173:0/4208218136 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63c005040 con 0x7fe654060280 2024-01-21T19:55:56.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.452+0000 7fe6537fe700 1 --2- 172.21.15.173:0/4208218136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe654060280 0x7fe6541088f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fe63c013510 tx=0x7fe63c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:56.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.452+0000 7fe64bfff700 1 -- 172.21.15.173:0/4208218136 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe63c009320 con 0x7fe654060280 2024-01-21T19:55:56.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.452+0000 7fe64bfff700 1 -- 172.21.15.173:0/4208218136 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe63c013f00 con 0x7fe654060280 2024-01-21T19:55:56.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.452+0000 7fe64bfff700 1 -- 172.21.15.173:0/4208218136 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe63c014dc0 con 0x7fe654060280 2024-01-21T19:55:56.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.452+0000 7fe659f1a700 1 -- 172.21.15.173:0/4208218136 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe654108fe0 con 0x7fe654060280 2024-01-21T19:55:56.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.453+0000 7fe659f1a700 1 -- 172.21.15.173:0/4208218136 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe654109470 con 0x7fe654060280 2024-01-21T19:55:56.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.453+0000 7fe64bfff700 1 -- 172.21.15.173:0/4208218136 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe63c014420 con 0x7fe654060280 2024-01-21T19:55:56.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.453+0000 7fe64bfff700 1 --2- 172.21.15.173:0/4208218136 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe64002ffb0 0x7fe640032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:56.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.453+0000 7fe64bfff700 1 -- 172.21.15.173:0/4208218136 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe63c0410b0 con 0x7fe654060280 2024-01-21T19:55:56.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.453+0000 7fe659f1a700 1 -- 172.21.15.173:0/4208218136 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe654045ad0 con 0x7fe654060280 2024-01-21T19:55:56.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.454+0000 7fe652ffd700 1 --2- 172.21.15.173:0/4208218136 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe64002ffb0 0x7fe640032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:56.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.454+0000 7fe652ffd700 1 --2- 172.21.15.173:0/4208218136 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe64002ffb0 0x7fe640032470 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fe6440096a0 tx=0x7fe644006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:56.450 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.458+0000 7fe64bfff700 1 -- 172.21.15.173:0/4208218136 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe63c0095f0 con 0x7fe654060280 2024-01-21T19:55:56.656 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.663+0000 7fe659f1a700 1 -- 172.21.15.173:0/4208218136 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6540fc040 con 0x7fe654060280 2024-01-21T19:55:56.656 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.664+0000 7fe64bfff700 1 -- 172.21.15.173:0/4208218136 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe63c01db70 con 0x7fe654060280 2024-01-21T19:55:56.656 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:56.657 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:56.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.666+0000 7fe659f1a700 1 -- 172.21.15.173:0/4208218136 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe64002ffb0 msgr2=0x7fe640032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:56.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.666+0000 7fe659f1a700 1 --2- 172.21.15.173:0/4208218136 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe64002ffb0 0x7fe640032470 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fe6440096a0 tx=0x7fe644006b40).stop 2024-01-21T19:55:56.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.666+0000 7fe659f1a700 1 -- 172.21.15.173:0/4208218136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe654060280 msgr2=0x7fe6541088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:56.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.666+0000 7fe659f1a700 1 --2- 172.21.15.173:0/4208218136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe654060280 0x7fe6541088f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fe63c013510 tx=0x7fe63c00b040).stop 2024-01-21T19:55:56.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.666+0000 7fe659f1a700 1 -- 172.21.15.173:0/4208218136 shutdown_connections 2024-01-21T19:55:56.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.666+0000 7fe659f1a700 1 --2- 172.21.15.173:0/4208218136 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe64002ffb0 0x7fe640032470 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:56.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.666+0000 7fe659f1a700 1 --2- 172.21.15.173:0/4208218136 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe654060280 0x7fe6541088f0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:56.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.666+0000 7fe659f1a700 1 -- 172.21.15.173:0/4208218136 >> 172.21.15.173:0/4208218136 conn(0x7fe6540f0b70 msgr2=0x7fe6540f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:56.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.666+0000 7fe659f1a700 1 -- 172.21.15.173:0/4208218136 shutdown_connections 2024-01-21T19:55:56.659 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:56.666+0000 7fe659f1a700 1 -- 172.21.15.173:0/4208218136 wait complete. 2024-01-21T19:55:56.659 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:55:57.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:56 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:55.004107+0000 mgr.smithi169.espgue (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:57.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:56 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:56.665932+0000 mon.smithi169 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.173:0/4208218136' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:55:58.050 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:55:58.050 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:55:58.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cephadm 2024-01-21T19:55:56.213490+0000 mgr.smithi169.espgue (mgr.14162) 229 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi169 ... 2024-01-21T19:55:58.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Verifying port 9100 ... 2024-01-21T19:55:58.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Non-zero exit code 1 from systemctl start ceph-df2470f2-b895-11ee-95b0-87774f69a715@node-exporter.smithi169 2024-01-21T19:55:58.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: systemctl: stderr Job for ceph-df2470f2-b895-11ee-95b0-87774f69a715@node-exporter.smithi169.service failed because a timeout was exceeded. 2024-01-21T19:55:58.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: systemctl: stderr See "systemctl status ceph-df2470f2-b895-11ee-95b0-87774f69a715@node-exporter.smithi169.service" and "journalctl -xe" for details. 2024-01-21T19:55:58.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Traceback (most recent call last): 2024-01-21T19:55:58.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T19:55:58.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: main() 2024-01-21T19:55:58.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T19:55:58.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: r = ctx.func(ctx) 2024-01-21T19:55:58.165 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T19:55:58.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: return func(ctx) 2024-01-21T19:55:58.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T19:55:58.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: ports=daemon_ports) 2024-01-21T19:55:58.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-21T19:55:58.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-21T19:55:58.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-21T19:55:58.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-21T19:55:58.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T19:55:58.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T19:55:58.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: RuntimeError: Failed command: systemctl start ceph-df2470f2-b895-11ee-95b0-87774f69a715@node-exporter.smithi169 2024-01-21T19:55:58.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Traceback (most recent call last): 2024-01-21T19:55:58.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T19:55:58.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: yield (conn, connr) 2024-01-21T19:55:58.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T19:55:58.166 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: code, '\n'.join(err))) 2024-01-21T19:55:58.167 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi169 ... 2024-01-21T19:55:58.167 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Verifying port 9100 ... 2024-01-21T19:55:58.167 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Non-zero exit code 1 from systemctl start ceph-df2470f2-b895-11ee-95b0-87774f69a715@node-exporter.smithi169 2024-01-21T19:55:58.167 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: systemctl: stderr Job for ceph-df2470f2-b895-11ee-95b0-87774f69a715@node-exporter.smithi169.service failed because a timeout was exceeded. 2024-01-21T19:55:58.167 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: systemctl: stderr See "systemctl status ceph-df2470f2-b895-11ee-95b0-87774f69a715@node-exporter.smithi169.service" and "journalctl -xe" for details. 2024-01-21T19:55:58.167 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: Traceback (most recent call last): 2024-01-21T19:55:58.167 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T19:55:58.167 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: main() 2024-01-21T19:55:58.167 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T19:55:58.167 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: r = ctx.func(ctx) 2024-01-21T19:55:58.167 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T19:55:58.167 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: return func(ctx) 2024-01-21T19:55:58.167 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T19:55:58.167 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: ports=daemon_ports) 2024-01-21T19:55:58.168 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-21T19:55:58.168 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-21T19:55:58.168 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-21T19:55:58.168 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-21T19:55:58.168 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: File "/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T19:55:58.168 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T19:55:58.168 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: RuntimeError: Failed command: systemctl start ceph-df2470f2-b895-11ee-95b0-87774f69a715@node-exporter.smithi169 2024-01-21T19:55:58.168 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:57 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cephadm 2024-01-21T19:55:56.218886+0000 mgr.smithi169.espgue (mgr.14162) 230 : cephadm [INF] Deploying daemon prometheus.smithi169 on smithi169 2024-01-21T19:55:59.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:57.004477+0000 mgr.smithi169.espgue (mgr.14162) 231 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:55:59.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.240+0000 7f8565e37700 1 -- 172.21.15.173:0/1889798497 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85600f47c0 msgr2=0x7f85600f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:59.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.240+0000 7f8565e37700 1 --2- 172.21.15.173:0/1889798497 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85600f47c0 0x7f85600f4be0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f8550009fb0 tx=0x7f8550009b30).stop 2024-01-21T19:55:59.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.240+0000 7f8565e37700 1 -- 172.21.15.173:0/1889798497 shutdown_connections 2024-01-21T19:55:59.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.240+0000 7f8565e37700 1 --2- 172.21.15.173:0/1889798497 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85600f47c0 0x7f85600f4be0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:59.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.240+0000 7f8565e37700 1 -- 172.21.15.173:0/1889798497 >> 172.21.15.173:0/1889798497 conn(0x7f85600f02e0 msgr2=0x7f85600f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:59.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.240+0000 7f8565e37700 1 -- 172.21.15.173:0/1889798497 shutdown_connections 2024-01-21T19:55:59.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.240+0000 7f8565e37700 1 -- 172.21.15.173:0/1889798497 wait complete. 2024-01-21T19:55:59.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.241+0000 7f8565e37700 1 Processor -- start 2024-01-21T19:55:59.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.241+0000 7f8565e37700 1 -- start start 2024-01-21T19:55:59.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.241+0000 7f8565e37700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85600f47c0 0x7f8560106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:59.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.241+0000 7f8565e37700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8560106bf0 con 0x7f85600f47c0 2024-01-21T19:55:59.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.242+0000 7f855f7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85600f47c0 0x7f8560106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:59.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.242+0000 7f855f7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85600f47c0 0x7f8560106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:37476/0 (socket says 172.21.15.173:37476) 2024-01-21T19:55:59.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.242+0000 7f855f7fe700 1 -- 172.21.15.173:0/1927830421 learned_addr learned my addr 172.21.15.173:0/1927830421 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:55:59.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.242+0000 7f855f7fe700 1 -- 172.21.15.173:0/1927830421 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8550005040 con 0x7f85600f47c0 2024-01-21T19:55:59.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.242+0000 7f855f7fe700 1 --2- 172.21.15.173:0/1927830421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85600f47c0 0x7f8560106700 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f8550013510 tx=0x7f855000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:59.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.242+0000 7f8557fff700 1 -- 172.21.15.173:0/1927830421 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8550009320 con 0x7f85600f47c0 2024-01-21T19:55:59.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.242+0000 7f8557fff700 1 -- 172.21.15.173:0/1927830421 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8550013f00 con 0x7f85600f47c0 2024-01-21T19:55:59.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.242+0000 7f8565e37700 1 -- 172.21.15.173:0/1927830421 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8560106df0 con 0x7f85600f47c0 2024-01-21T19:55:59.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.242+0000 7f8557fff700 1 -- 172.21.15.173:0/1927830421 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8550014d90 con 0x7f85600f47c0 2024-01-21T19:55:59.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.242+0000 7f8565e37700 1 -- 172.21.15.173:0/1927830421 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8560107280 con 0x7f85600f47c0 2024-01-21T19:55:59.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.243+0000 7f8557fff700 1 -- 172.21.15.173:0/1927830421 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f85500263a0 con 0x7f85600f47c0 2024-01-21T19:55:59.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.243+0000 7f8565e37700 1 -- 172.21.15.173:0/1927830421 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8560045ad0 con 0x7f85600f47c0 2024-01-21T19:55:59.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.243+0000 7f8557fff700 1 --2- 172.21.15.173:0/1927830421 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f854c02ff60 0x7f854c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:55:59.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.243+0000 7f8557fff700 1 -- 172.21.15.173:0/1927830421 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8550041320 con 0x7f85600f47c0 2024-01-21T19:55:59.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.243+0000 7f855effd700 1 --2- 172.21.15.173:0/1927830421 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f854c02ff60 0x7f854c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:55:59.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.244+0000 7f855effd700 1 --2- 172.21.15.173:0/1927830421 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f854c02ff60 0x7f854c032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f85480096a0 tx=0x7f8548006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:55:59.240 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.248+0000 7f8557fff700 1 -- 172.21.15.173:0/1927830421 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f85500126f0 con 0x7f85600f47c0 2024-01-21T19:55:59.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.451+0000 7f8565e37700 1 -- 172.21.15.173:0/1927830421 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8560108130 con 0x7f85600f47c0 2024-01-21T19:55:59.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.453+0000 7f8557fff700 1 -- 172.21.15.173:0/1927830421 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8550012330 con 0x7f85600f47c0 2024-01-21T19:55:59.444 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:55:59.444 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:55:59.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.453+0000 7f8565e37700 1 -- 172.21.15.173:0/1927830421 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f854c02ff60 msgr2=0x7f854c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:59.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.453+0000 7f8565e37700 1 --2- 172.21.15.173:0/1927830421 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f854c02ff60 0x7f854c032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f85480096a0 tx=0x7f8548006b40).stop 2024-01-21T19:55:59.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.453+0000 7f8565e37700 1 -- 172.21.15.173:0/1927830421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85600f47c0 msgr2=0x7f8560106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:55:59.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.453+0000 7f8565e37700 1 --2- 172.21.15.173:0/1927830421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85600f47c0 0x7f8560106700 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f8550013510 tx=0x7f855000b040).stop 2024-01-21T19:55:59.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.453+0000 7f8565e37700 1 -- 172.21.15.173:0/1927830421 shutdown_connections 2024-01-21T19:55:59.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.453+0000 7f8565e37700 1 --2- 172.21.15.173:0/1927830421 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f854c02ff60 0x7f854c032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:59.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.453+0000 7f8565e37700 1 --2- 172.21.15.173:0/1927830421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f85600f47c0 0x7f8560106700 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:55:59.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.453+0000 7f8565e37700 1 -- 172.21.15.173:0/1927830421 >> 172.21.15.173:0/1927830421 conn(0x7f85600f02e0 msgr2=0x7f85600f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:55:59.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.454+0000 7f8565e37700 1 -- 172.21.15.173:0/1927830421 shutdown_connections 2024-01-21T19:55:59.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:55:59.454+0000 7f8565e37700 1 -- 172.21.15.173:0/1927830421 wait complete. 2024-01-21T19:55:59.447 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:00.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:55:59 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:55:59.453694+0000 mon.smithi169 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.173:0/1927830421' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:00.804 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:00.805 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:56:01.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:00 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:55:59.004843+0000 mgr.smithi169.espgue (mgr.14162) 232 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:02.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.022+0000 7f91f4555700 1 -- 172.21.15.173:0/2342530531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91ec0f5410 msgr2=0x7f91ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:02.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.022+0000 7f91f4555700 1 --2- 172.21.15.173:0/2342530531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91ec0f5410 0x7f91ec0f5830 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f91e8009fb0 tx=0x7f91e8009b30).stop 2024-01-21T19:56:02.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.022+0000 7f91f4555700 1 -- 172.21.15.173:0/2342530531 shutdown_connections 2024-01-21T19:56:02.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.022+0000 7f91f4555700 1 --2- 172.21.15.173:0/2342530531 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91ec0f5410 0x7f91ec0f5830 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:02.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.022+0000 7f91f4555700 1 -- 172.21.15.173:0/2342530531 >> 172.21.15.173:0/2342530531 conn(0x7f91ec0f0b70 msgr2=0x7f91ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:02.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.022+0000 7f91f4555700 1 -- 172.21.15.173:0/2342530531 shutdown_connections 2024-01-21T19:56:02.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.022+0000 7f91f4555700 1 -- 172.21.15.173:0/2342530531 wait complete. 2024-01-21T19:56:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.023+0000 7f91f4555700 1 Processor -- start 2024-01-21T19:56:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.023+0000 7f91f4555700 1 -- start start 2024-01-21T19:56:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.023+0000 7f91f4555700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91ec0f5410 0x7f91ec105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.023+0000 7f91f4555700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec105b20 con 0x7f91ec0f5410 2024-01-21T19:56:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.023+0000 7f91f22f1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91ec0f5410 0x7f91ec105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.023+0000 7f91f22f1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91ec0f5410 0x7f91ec105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:37492/0 (socket says 172.21.15.173:37492) 2024-01-21T19:56:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.023+0000 7f91f22f1700 1 -- 172.21.15.173:0/631096995 learned_addr learned my addr 172.21.15.173:0/631096995 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:56:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.023+0000 7f91f22f1700 1 -- 172.21.15.173:0/631096995 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e8005040 con 0x7f91ec0f5410 2024-01-21T19:56:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.023+0000 7f91f22f1700 1 --2- 172.21.15.173:0/631096995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91ec0f5410 0x7f91ec105630 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f91e8013510 tx=0x7f91e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.023+0000 7f91e2ffd700 1 -- 172.21.15.173:0/631096995 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91e8009320 con 0x7f91ec0f5410 2024-01-21T19:56:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.024+0000 7f91f4555700 1 -- 172.21.15.173:0/631096995 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91ec105d20 con 0x7f91ec0f5410 2024-01-21T19:56:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.024+0000 7f91e2ffd700 1 -- 172.21.15.173:0/631096995 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91e8013f00 con 0x7f91ec0f5410 2024-01-21T19:56:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.024+0000 7f91e2ffd700 1 -- 172.21.15.173:0/631096995 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91e8014d90 con 0x7f91ec0f5410 2024-01-21T19:56:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.024+0000 7f91f4555700 1 -- 172.21.15.173:0/631096995 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ec1061b0 con 0x7f91ec0f5410 2024-01-21T19:56:02.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.024+0000 7f91e2ffd700 1 -- 172.21.15.173:0/631096995 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f91e800f410 con 0x7f91ec0f5410 2024-01-21T19:56:02.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.025+0000 7f91f4555700 1 -- 172.21.15.173:0/631096995 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d4004fa0 con 0x7f91ec0f5410 2024-01-21T19:56:02.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.025+0000 7f91e2ffd700 1 --2- 172.21.15.173:0/631096995 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f91d802ff60 0x7f91d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:02.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.025+0000 7f91e2ffd700 1 -- 172.21.15.173:0/631096995 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f91e8043030 con 0x7f91ec0f5410 2024-01-21T19:56:02.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.025+0000 7f91f1af0700 1 --2- 172.21.15.173:0/631096995 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f91d802ff60 0x7f91d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:02.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.025+0000 7f91f1af0700 1 --2- 172.21.15.173:0/631096995 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f91d802ff60 0x7f91d8032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f91dc0096a0 tx=0x7f91dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:02.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.030+0000 7f91e2ffd700 1 -- 172.21.15.173:0/631096995 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f91e80126f0 con 0x7f91ec0f5410 2024-01-21T19:56:02.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.233+0000 7f91f4555700 1 -- 172.21.15.173:0/631096995 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f91d4005670 con 0x7f91ec0f5410 2024-01-21T19:56:02.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.234+0000 7f91e2ffd700 1 -- 172.21.15.173:0/631096995 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f91e8012330 con 0x7f91ec0f5410 2024-01-21T19:56:02.226 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:56:02.227 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:56:02.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.235+0000 7f91f4555700 1 -- 172.21.15.173:0/631096995 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f91d802ff60 msgr2=0x7f91d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:02.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.235+0000 7f91f4555700 1 --2- 172.21.15.173:0/631096995 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f91d802ff60 0x7f91d8032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f91dc0096a0 tx=0x7f91dc006b40).stop 2024-01-21T19:56:02.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.235+0000 7f91f4555700 1 -- 172.21.15.173:0/631096995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91ec0f5410 msgr2=0x7f91ec105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:02.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.235+0000 7f91f4555700 1 --2- 172.21.15.173:0/631096995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91ec0f5410 0x7f91ec105630 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f91e8013510 tx=0x7f91e800b040).stop 2024-01-21T19:56:02.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.236+0000 7f91f4555700 1 -- 172.21.15.173:0/631096995 shutdown_connections 2024-01-21T19:56:02.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.236+0000 7f91f4555700 1 --2- 172.21.15.173:0/631096995 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f91d802ff60 0x7f91d8032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:02.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.236+0000 7f91f4555700 1 --2- 172.21.15.173:0/631096995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f91ec0f5410 0x7f91ec105630 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:02.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.236+0000 7f91f4555700 1 -- 172.21.15.173:0/631096995 >> 172.21.15.173:0/631096995 conn(0x7f91ec0f0b70 msgr2=0x7f91ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:02.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.236+0000 7f91f4555700 1 -- 172.21.15.173:0/631096995 shutdown_connections 2024-01-21T19:56:02.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:02.236+0000 7f91f4555700 1 -- 172.21.15.173:0/631096995 wait complete. 2024-01-21T19:56:02.229 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:03.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:02 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:01.005230+0000 mgr.smithi169.espgue (mgr.14162) 233 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:03.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:02 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:02.235883+0000 mon.smithi169 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.173:0/631096995' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:03.619 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:03.620 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:56:04.779 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.786+0000 7ffa250ef700 1 -- 172.21.15.173:0/1491423328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa200f6510 msgr2=0x7ffa200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:04.779 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.786+0000 7ffa250ef700 1 --2- 172.21.15.173:0/1491423328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa200f6510 0x7ffa200f6930 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7ffa14009fb0 tx=0x7ffa14009b30).stop 2024-01-21T19:56:04.779 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.786+0000 7ffa250ef700 1 -- 172.21.15.173:0/1491423328 shutdown_connections 2024-01-21T19:56:04.779 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.786+0000 7ffa250ef700 1 --2- 172.21.15.173:0/1491423328 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa200f6510 0x7ffa200f6930 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:04.780 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.786+0000 7ffa250ef700 1 -- 172.21.15.173:0/1491423328 >> 172.21.15.173:0/1491423328 conn(0x7ffa200f1c70 msgr2=0x7ffa200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:04.780 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.786+0000 7ffa250ef700 1 -- 172.21.15.173:0/1491423328 shutdown_connections 2024-01-21T19:56:04.780 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.786+0000 7ffa250ef700 1 -- 172.21.15.173:0/1491423328 wait complete. 2024-01-21T19:56:04.780 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.787+0000 7ffa250ef700 1 Processor -- start 2024-01-21T19:56:04.780 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.787+0000 7ffa250ef700 1 -- start start 2024-01-21T19:56:04.780 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.787+0000 7ffa250ef700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa200f6510 0x7ffa2006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:04.780 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.787+0000 7ffa250ef700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa20069e20 con 0x7ffa200f6510 2024-01-21T19:56:04.780 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.787+0000 7ffa1ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa200f6510 0x7ffa2006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:04.780 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.787+0000 7ffa1ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa200f6510 0x7ffa2006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:44972/0 (socket says 172.21.15.173:44972) 2024-01-21T19:56:04.780 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.787+0000 7ffa1ed9d700 1 -- 172.21.15.173:0/2085142579 learned_addr learned my addr 172.21.15.173:0/2085142579 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:56:04.780 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.788+0000 7ffa1ed9d700 1 -- 172.21.15.173:0/2085142579 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa14005040 con 0x7ffa200f6510 2024-01-21T19:56:04.780 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.788+0000 7ffa1ed9d700 1 --2- 172.21.15.173:0/2085142579 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa200f6510 0x7ffa2006b9a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7ffa14013510 tx=0x7ffa1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:04.780 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.788+0000 7ffa077fe700 1 -- 172.21.15.173:0/2085142579 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa14009320 con 0x7ffa200f6510 2024-01-21T19:56:04.780 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.788+0000 7ffa250ef700 1 -- 172.21.15.173:0/2085142579 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa20069fc0 con 0x7ffa200f6510 2024-01-21T19:56:04.780 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.788+0000 7ffa250ef700 1 -- 172.21.15.173:0/2085142579 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2006a450 con 0x7ffa200f6510 2024-01-21T19:56:04.781 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.788+0000 7ffa077fe700 1 -- 172.21.15.173:0/2085142579 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa14013f00 con 0x7ffa200f6510 2024-01-21T19:56:04.781 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.788+0000 7ffa077fe700 1 -- 172.21.15.173:0/2085142579 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa14014d60 con 0x7ffa200f6510 2024-01-21T19:56:04.784 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.789+0000 7ffa077fe700 1 -- 172.21.15.173:0/2085142579 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffa14014420 con 0x7ffa200f6510 2024-01-21T19:56:04.784 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.789+0000 7ffa250ef700 1 -- 172.21.15.173:0/2085142579 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa20059100 con 0x7ffa200f6510 2024-01-21T19:56:04.784 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.789+0000 7ffa077fe700 1 --2- 172.21.15.173:0/2085142579 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ffa0802ff60 0x7ffa08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:04.784 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.789+0000 7ffa077fe700 1 -- 172.21.15.173:0/2085142579 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffa14041390 con 0x7ffa200f6510 2024-01-21T19:56:04.784 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.789+0000 7ffa1e59c700 1 --2- 172.21.15.173:0/2085142579 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ffa0802ff60 0x7ffa08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:04.784 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.789+0000 7ffa1e59c700 1 --2- 172.21.15.173:0/2085142579 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ffa0802ff60 0x7ffa08032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7ffa100096a0 tx=0x7ffa10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:04.785 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:04.793+0000 7ffa077fe700 1 -- 172.21.15.173:0/2085142579 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffa14012b20 con 0x7ffa200f6510 2024-01-21T19:56:04.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:05.001+0000 7ffa250ef700 1 -- 172.21.15.173:0/2085142579 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffa200fbf60 con 0x7ffa200f6510 2024-01-21T19:56:04.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:05.002+0000 7ffa077fe700 1 -- 172.21.15.173:0/2085142579 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffa14012b20 con 0x7ffa200f6510 2024-01-21T19:56:04.993 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:56:04.993 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:56:04.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:05.003+0000 7ffa250ef700 1 -- 172.21.15.173:0/2085142579 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ffa0802ff60 msgr2=0x7ffa08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:04.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:05.003+0000 7ffa250ef700 1 --2- 172.21.15.173:0/2085142579 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ffa0802ff60 0x7ffa08032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7ffa100096a0 tx=0x7ffa10006b40).stop 2024-01-21T19:56:04.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:05.003+0000 7ffa250ef700 1 -- 172.21.15.173:0/2085142579 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa200f6510 msgr2=0x7ffa2006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:04.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:05.003+0000 7ffa250ef700 1 --2- 172.21.15.173:0/2085142579 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa200f6510 0x7ffa2006b9a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7ffa14013510 tx=0x7ffa1400b040).stop 2024-01-21T19:56:04.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:05.003+0000 7ffa250ef700 1 -- 172.21.15.173:0/2085142579 shutdown_connections 2024-01-21T19:56:04.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:05.003+0000 7ffa250ef700 1 --2- 172.21.15.173:0/2085142579 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ffa0802ff60 0x7ffa08032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:04.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:05.003+0000 7ffa250ef700 1 --2- 172.21.15.173:0/2085142579 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffa200f6510 0x7ffa2006b9a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:04.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:05.003+0000 7ffa250ef700 1 -- 172.21.15.173:0/2085142579 >> 172.21.15.173:0/2085142579 conn(0x7ffa200f1c70 msgr2=0x7ffa200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:04.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:05.004+0000 7ffa250ef700 1 -- 172.21.15.173:0/2085142579 shutdown_connections 2024-01-21T19:56:04.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:05.004+0000 7ffa250ef700 1 -- 172.21.15.173:0/2085142579 wait complete. 2024-01-21T19:56:04.996 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:05.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:04 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:03.005602+0000 mgr.smithi169.espgue (mgr.14162) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:06.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:05 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:05.002600+0000 mon.smithi169 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.173:0/2085142579' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:06.386 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:06.386 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:56:07.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:06 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:05.006008+0000 mgr.smithi169.espgue (mgr.14162) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:07.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.630+0000 7f10c26d0700 1 -- 172.21.15.173:0/913886854 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10bc0f5410 msgr2=0x7f10bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:07.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.630+0000 7f10c26d0700 1 --2- 172.21.15.173:0/913886854 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10bc0f5410 0x7f10bc0f5830 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f10ac009fb0 tx=0x7f10ac009b30).stop 2024-01-21T19:56:07.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.630+0000 7f10c26d0700 1 -- 172.21.15.173:0/913886854 shutdown_connections 2024-01-21T19:56:07.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.630+0000 7f10c26d0700 1 --2- 172.21.15.173:0/913886854 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10bc0f5410 0x7f10bc0f5830 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:07.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.630+0000 7f10c26d0700 1 -- 172.21.15.173:0/913886854 >> 172.21.15.173:0/913886854 conn(0x7f10bc0f0b70 msgr2=0x7f10bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:07.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.630+0000 7f10c26d0700 1 -- 172.21.15.173:0/913886854 shutdown_connections 2024-01-21T19:56:07.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.630+0000 7f10c26d0700 1 -- 172.21.15.173:0/913886854 wait complete. 2024-01-21T19:56:07.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.630+0000 7f10c26d0700 1 Processor -- start 2024-01-21T19:56:07.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.630+0000 7f10c26d0700 1 -- start start 2024-01-21T19:56:07.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.630+0000 7f10c26d0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10bc0f5410 0x7f10bc0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:07.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.630+0000 7f10c26d0700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc0fb820 con 0x7f10bc0f5410 2024-01-21T19:56:07.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.631+0000 7f10bbfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10bc0f5410 0x7f10bc0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:07.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.631+0000 7f10bbfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10bc0f5410 0x7f10bc0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:44980/0 (socket says 172.21.15.173:44980) 2024-01-21T19:56:07.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.631+0000 7f10bbfff700 1 -- 172.21.15.173:0/2599361600 learned_addr learned my addr 172.21.15.173:0/2599361600 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:56:07.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.631+0000 7f10bbfff700 1 -- 172.21.15.173:0/2599361600 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ac005040 con 0x7f10bc0f5410 2024-01-21T19:56:07.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.631+0000 7f10bbfff700 1 --2- 172.21.15.173:0/2599361600 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10bc0f5410 0x7f10bc0fd3a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f10ac013510 tx=0x7f10ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:07.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.631+0000 7f10b8ff9700 1 -- 172.21.15.173:0/2599361600 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10ac009320 con 0x7f10bc0f5410 2024-01-21T19:56:07.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.631+0000 7f10b8ff9700 1 -- 172.21.15.173:0/2599361600 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f10ac013f00 con 0x7f10bc0f5410 2024-01-21T19:56:07.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.631+0000 7f10c26d0700 1 -- 172.21.15.173:0/2599361600 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10bc0fb9c0 con 0x7f10bc0f5410 2024-01-21T19:56:07.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.631+0000 7f10b8ff9700 1 -- 172.21.15.173:0/2599361600 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10ac014d90 con 0x7f10bc0f5410 2024-01-21T19:56:07.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.631+0000 7f10c26d0700 1 -- 172.21.15.173:0/2599361600 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10bc0fbe50 con 0x7f10bc0f5410 2024-01-21T19:56:07.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.632+0000 7f10b8ff9700 1 -- 172.21.15.173:0/2599361600 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f10ac0095f0 con 0x7f10bc0f5410 2024-01-21T19:56:07.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.632+0000 7f10c26d0700 1 -- 172.21.15.173:0/2599361600 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10bc059100 con 0x7f10bc0f5410 2024-01-21T19:56:07.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.632+0000 7f10b8ff9700 1 --2- 172.21.15.173:0/2599361600 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f10a402ff60 0x7f10a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:07.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.632+0000 7f10b8ff9700 1 -- 172.21.15.173:0/2599361600 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f10ac041000 con 0x7f10bc0f5410 2024-01-21T19:56:07.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.633+0000 7f10bb7fe700 1 --2- 172.21.15.173:0/2599361600 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f10a402ff60 0x7f10a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:07.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.633+0000 7f10bb7fe700 1 --2- 172.21.15.173:0/2599361600 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f10a402ff60 0x7f10a4032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f10b00096a0 tx=0x7f10b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:07.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.637+0000 7f10b8ff9700 1 -- 172.21.15.173:0/2599361600 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f10ac014420 con 0x7f10bc0f5410 2024-01-21T19:56:07.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.841+0000 7f10c26d0700 1 -- 172.21.15.173:0/2599361600 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f10bc0fae60 con 0x7f10bc0f5410 2024-01-21T19:56:07.833 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.841+0000 7f10b8ff9700 1 -- 172.21.15.173:0/2599361600 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f10ac01d7f0 con 0x7f10bc0f5410 2024-01-21T19:56:07.833 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:56:07.833 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:56:07.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.843+0000 7f10c26d0700 1 -- 172.21.15.173:0/2599361600 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f10a402ff60 msgr2=0x7f10a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:07.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.843+0000 7f10c26d0700 1 --2- 172.21.15.173:0/2599361600 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f10a402ff60 0x7f10a4032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f10b00096a0 tx=0x7f10b0006b40).stop 2024-01-21T19:56:07.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.843+0000 7f10c26d0700 1 -- 172.21.15.173:0/2599361600 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10bc0f5410 msgr2=0x7f10bc0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:07.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.843+0000 7f10c26d0700 1 --2- 172.21.15.173:0/2599361600 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10bc0f5410 0x7f10bc0fd3a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f10ac013510 tx=0x7f10ac00b040).stop 2024-01-21T19:56:07.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.843+0000 7f10c26d0700 1 -- 172.21.15.173:0/2599361600 shutdown_connections 2024-01-21T19:56:07.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.843+0000 7f10c26d0700 1 --2- 172.21.15.173:0/2599361600 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f10a402ff60 0x7f10a4032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:07.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.843+0000 7f10c26d0700 1 --2- 172.21.15.173:0/2599361600 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10bc0f5410 0x7f10bc0fd3a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:07.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.843+0000 7f10c26d0700 1 -- 172.21.15.173:0/2599361600 >> 172.21.15.173:0/2599361600 conn(0x7f10bc0f0b70 msgr2=0x7f10bc05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:07.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.843+0000 7f10c26d0700 1 -- 172.21.15.173:0/2599361600 shutdown_connections 2024-01-21T19:56:07.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:07.843+0000 7f10c26d0700 1 -- 172.21.15.173:0/2599361600 wait complete. 2024-01-21T19:56:07.835 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:09.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:08 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:07.006291+0000 mgr.smithi169.espgue (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:09.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:08 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:07.842282+0000 mon.smithi169 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.173:0/2599361600' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:09.219 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:09.219 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:56:10.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.448+0000 7f70edbd8700 1 -- 172.21.15.173:0/1558728371 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70e80f5410 msgr2=0x7f70e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:10.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.448+0000 7f70edbd8700 1 --2- 172.21.15.173:0/1558728371 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70e80f5410 0x7f70e80f5830 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f70d8009fb0 tx=0x7f70d8009b30).stop 2024-01-21T19:56:10.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.448+0000 7f70edbd8700 1 -- 172.21.15.173:0/1558728371 shutdown_connections 2024-01-21T19:56:10.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.448+0000 7f70edbd8700 1 --2- 172.21.15.173:0/1558728371 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70e80f5410 0x7f70e80f5830 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:10.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.448+0000 7f70edbd8700 1 -- 172.21.15.173:0/1558728371 >> 172.21.15.173:0/1558728371 conn(0x7f70e80f0b70 msgr2=0x7f70e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:10.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.448+0000 7f70edbd8700 1 -- 172.21.15.173:0/1558728371 shutdown_connections 2024-01-21T19:56:10.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.448+0000 7f70edbd8700 1 -- 172.21.15.173:0/1558728371 wait complete. 2024-01-21T19:56:10.441 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.449+0000 7f70edbd8700 1 Processor -- start 2024-01-21T19:56:10.441 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.449+0000 7f70edbd8700 1 -- start start 2024-01-21T19:56:10.441 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.449+0000 7f70edbd8700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70e80f5410 0x7f70e8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:10.441 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.449+0000 7f70edbd8700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e8105b20 con 0x7f70e80f5410 2024-01-21T19:56:10.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.449+0000 7f70e77fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70e80f5410 0x7f70e8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:10.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.449+0000 7f70e77fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70e80f5410 0x7f70e8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:44996/0 (socket says 172.21.15.173:44996) 2024-01-21T19:56:10.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.449+0000 7f70e77fe700 1 -- 172.21.15.173:0/2541646525 learned_addr learned my addr 172.21.15.173:0/2541646525 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:56:10.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.450+0000 7f70e77fe700 1 -- 172.21.15.173:0/2541646525 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d8005040 con 0x7f70e80f5410 2024-01-21T19:56:10.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.450+0000 7f70e77fe700 1 --2- 172.21.15.173:0/2541646525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70e80f5410 0x7f70e8105630 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f70d8013510 tx=0x7f70d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:10.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.450+0000 7f70e4ff9700 1 -- 172.21.15.173:0/2541646525 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70d8009320 con 0x7f70e80f5410 2024-01-21T19:56:10.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.450+0000 7f70edbd8700 1 -- 172.21.15.173:0/2541646525 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e8105d20 con 0x7f70e80f5410 2024-01-21T19:56:10.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.450+0000 7f70e4ff9700 1 -- 172.21.15.173:0/2541646525 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f70d8013f00 con 0x7f70e80f5410 2024-01-21T19:56:10.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.450+0000 7f70e4ff9700 1 -- 172.21.15.173:0/2541646525 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70d8014d90 con 0x7f70e80f5410 2024-01-21T19:56:10.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.450+0000 7f70edbd8700 1 -- 172.21.15.173:0/2541646525 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e81061b0 con 0x7f70e80f5410 2024-01-21T19:56:10.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.451+0000 7f70e4ff9700 1 -- 172.21.15.173:0/2541646525 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f70d8014420 con 0x7f70e80f5410 2024-01-21T19:56:10.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.451+0000 7f70e4ff9700 1 --2- 172.21.15.173:0/2541646525 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f70d402ff60 0x7f70d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:10.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.451+0000 7f70edbd8700 1 -- 172.21.15.173:0/2541646525 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e8045ad0 con 0x7f70e80f5410 2024-01-21T19:56:10.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.451+0000 7f70e4ff9700 1 -- 172.21.15.173:0/2541646525 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f70d8041000 con 0x7f70e80f5410 2024-01-21T19:56:10.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.451+0000 7f70dffff700 1 --2- 172.21.15.173:0/2541646525 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f70d402ff60 0x7f70d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:10.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.451+0000 7f70dffff700 1 --2- 172.21.15.173:0/2541646525 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f70d402ff60 0x7f70d4032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f70d00096a0 tx=0x7f70d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:10.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.455+0000 7f70e4ff9700 1 -- 172.21.15.173:0/2541646525 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f70d806c2b0 con 0x7f70e80f5410 2024-01-21T19:56:10.651 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.659+0000 7f70edbd8700 1 -- 172.21.15.173:0/2541646525 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f70e80faf50 con 0x7f70e80f5410 2024-01-21T19:56:10.651 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.660+0000 7f70e4ff9700 1 -- 172.21.15.173:0/2541646525 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f70d802d090 con 0x7f70e80f5410 2024-01-21T19:56:10.652 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:56:10.652 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:56:10.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.662+0000 7f70edbd8700 1 -- 172.21.15.173:0/2541646525 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f70d402ff60 msgr2=0x7f70d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:10.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.662+0000 7f70edbd8700 1 --2- 172.21.15.173:0/2541646525 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f70d402ff60 0x7f70d4032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f70d00096a0 tx=0x7f70d0006b40).stop 2024-01-21T19:56:10.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.662+0000 7f70edbd8700 1 -- 172.21.15.173:0/2541646525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70e80f5410 msgr2=0x7f70e8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:10.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.662+0000 7f70edbd8700 1 --2- 172.21.15.173:0/2541646525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70e80f5410 0x7f70e8105630 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f70d8013510 tx=0x7f70d800b040).stop 2024-01-21T19:56:10.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.662+0000 7f70edbd8700 1 -- 172.21.15.173:0/2541646525 shutdown_connections 2024-01-21T19:56:10.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.662+0000 7f70edbd8700 1 --2- 172.21.15.173:0/2541646525 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f70d402ff60 0x7f70d4032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:10.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.662+0000 7f70edbd8700 1 --2- 172.21.15.173:0/2541646525 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f70e80f5410 0x7f70e8105630 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:10.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.662+0000 7f70edbd8700 1 -- 172.21.15.173:0/2541646525 >> 172.21.15.173:0/2541646525 conn(0x7f70e80f0b70 msgr2=0x7f70e80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:10.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.662+0000 7f70edbd8700 1 -- 172.21.15.173:0/2541646525 shutdown_connections 2024-01-21T19:56:10.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:10.662+0000 7f70edbd8700 1 -- 172.21.15.173:0/2541646525 wait complete. 2024-01-21T19:56:10.654 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:11.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:10 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:09.006544+0000 mgr.smithi169.espgue (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:11.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:10 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:10.660881+0000 mon.smithi169 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.173:0/2541646525' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:12.041 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:12.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:56:13.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:12 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:11.006894+0000 mgr.smithi169.espgue (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:13.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.202+0000 7f371628c700 1 -- 172.21.15.173:0/632557610 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3710060280 msgr2=0x7f37100f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:13.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.202+0000 7f371628c700 1 --2- 172.21.15.173:0/632557610 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3710060280 0x7f37100f90f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f3700009fb0 tx=0x7f3700009b30).stop 2024-01-21T19:56:13.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.202+0000 7f371628c700 1 -- 172.21.15.173:0/632557610 shutdown_connections 2024-01-21T19:56:13.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.202+0000 7f371628c700 1 --2- 172.21.15.173:0/632557610 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3710060280 0x7f37100f90f0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:13.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.202+0000 7f371628c700 1 -- 172.21.15.173:0/632557610 >> 172.21.15.173:0/632557610 conn(0x7f37100f0b70 msgr2=0x7f37100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:13.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.202+0000 7f371628c700 1 -- 172.21.15.173:0/632557610 shutdown_connections 2024-01-21T19:56:13.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.202+0000 7f371628c700 1 -- 172.21.15.173:0/632557610 wait complete. 2024-01-21T19:56:13.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.203+0000 7f371628c700 1 Processor -- start 2024-01-21T19:56:13.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.203+0000 7f371628c700 1 -- start start 2024-01-21T19:56:13.197 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.203+0000 7f371628c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3710060280 0x7f37101066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:13.197 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.203+0000 7f371628c700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3710106bd0 con 0x7f3710060280 2024-01-21T19:56:13.197 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.204+0000 7f370ffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3710060280 0x7f37101066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:13.197 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.204+0000 7f370ffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3710060280 0x7f37101066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:45020/0 (socket says 172.21.15.173:45020) 2024-01-21T19:56:13.197 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.204+0000 7f370ffff700 1 -- 172.21.15.173:0/2432015920 learned_addr learned my addr 172.21.15.173:0/2432015920 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:56:13.197 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.204+0000 7f370ffff700 1 -- 172.21.15.173:0/2432015920 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3700005040 con 0x7f3710060280 2024-01-21T19:56:13.197 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.204+0000 7f370ffff700 1 --2- 172.21.15.173:0/2432015920 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3710060280 0x7f37101066e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f3700013510 tx=0x7f370000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:13.197 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.204+0000 7f370cff9700 1 -- 172.21.15.173:0/2432015920 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3700009320 con 0x7f3710060280 2024-01-21T19:56:13.197 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.204+0000 7f371628c700 1 -- 172.21.15.173:0/2432015920 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3710106dd0 con 0x7f3710060280 2024-01-21T19:56:13.197 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.204+0000 7f370cff9700 1 -- 172.21.15.173:0/2432015920 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3700013f00 con 0x7f3710060280 2024-01-21T19:56:13.197 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.204+0000 7f370cff9700 1 -- 172.21.15.173:0/2432015920 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3700014dc0 con 0x7f3710060280 2024-01-21T19:56:13.197 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.204+0000 7f371628c700 1 -- 172.21.15.173:0/2432015920 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3710107260 con 0x7f3710060280 2024-01-21T19:56:13.198 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.205+0000 7f370cff9700 1 -- 172.21.15.173:0/2432015920 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f37000095f0 con 0x7f3710060280 2024-01-21T19:56:13.198 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.205+0000 7f371628c700 1 -- 172.21.15.173:0/2432015920 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3710045ad0 con 0x7f3710060280 2024-01-21T19:56:13.198 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.205+0000 7f370cff9700 1 --2- 172.21.15.173:0/2432015920 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f36f802ff60 0x7f36f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:13.198 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.205+0000 7f370cff9700 1 -- 172.21.15.173:0/2432015920 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f37000410e0 con 0x7f3710060280 2024-01-21T19:56:13.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.206+0000 7f370f7fe700 1 --2- 172.21.15.173:0/2432015920 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f36f802ff60 0x7f36f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:13.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.206+0000 7f370f7fe700 1 --2- 172.21.15.173:0/2432015920 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f36f802ff60 0x7f36f8032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f37040096a0 tx=0x7f3704006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:13.201 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.210+0000 7f370cff9700 1 -- 172.21.15.173:0/2432015920 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3700014420 con 0x7f3710060280 2024-01-21T19:56:13.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.417+0000 7f371628c700 1 -- 172.21.15.173:0/2432015920 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3710108090 con 0x7f3710060280 2024-01-21T19:56:13.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.418+0000 7f370cff9700 1 -- 172.21.15.173:0/2432015920 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f370001c920 con 0x7f3710060280 2024-01-21T19:56:13.409 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:56:13.409 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:56:13.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.419+0000 7f371628c700 1 -- 172.21.15.173:0/2432015920 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f36f802ff60 msgr2=0x7f36f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:13.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.419+0000 7f371628c700 1 --2- 172.21.15.173:0/2432015920 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f36f802ff60 0x7f36f8032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f37040096a0 tx=0x7f3704006b40).stop 2024-01-21T19:56:13.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.419+0000 7f371628c700 1 -- 172.21.15.173:0/2432015920 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3710060280 msgr2=0x7f37101066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:13.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.419+0000 7f371628c700 1 --2- 172.21.15.173:0/2432015920 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3710060280 0x7f37101066e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f3700013510 tx=0x7f370000b040).stop 2024-01-21T19:56:13.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.419+0000 7f371628c700 1 -- 172.21.15.173:0/2432015920 shutdown_connections 2024-01-21T19:56:13.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.420+0000 7f371628c700 1 --2- 172.21.15.173:0/2432015920 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f36f802ff60 0x7f36f8032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:13.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.420+0000 7f371628c700 1 --2- 172.21.15.173:0/2432015920 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3710060280 0x7f37101066e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:13.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.420+0000 7f371628c700 1 -- 172.21.15.173:0/2432015920 >> 172.21.15.173:0/2432015920 conn(0x7f37100f0b70 msgr2=0x7f37100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:13.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.420+0000 7f371628c700 1 -- 172.21.15.173:0/2432015920 shutdown_connections 2024-01-21T19:56:13.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:13.420+0000 7f371628c700 1 -- 172.21.15.173:0/2432015920 wait complete. 2024-01-21T19:56:13.412 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:14.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:13 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:13.418537+0000 mon.smithi169 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.173:0/2432015920' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:14.796 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:14.796 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:56:15.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:14 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:13.007189+0000 mgr.smithi169.espgue (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:15.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.000+0000 7f3ab1765700 1 -- 172.21.15.173:0/2809715688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3aac0f5410 msgr2=0x7f3aac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:15.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.000+0000 7f3ab1765700 1 --2- 172.21.15.173:0/2809715688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3aac0f5410 0x7f3aac0f5830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f3a94009fb0 tx=0x7f3a94009b30).stop 2024-01-21T19:56:15.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.000+0000 7f3ab1765700 1 -- 172.21.15.173:0/2809715688 shutdown_connections 2024-01-21T19:56:15.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.000+0000 7f3ab1765700 1 --2- 172.21.15.173:0/2809715688 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3aac0f5410 0x7f3aac0f5830 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:15.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.000+0000 7f3ab1765700 1 -- 172.21.15.173:0/2809715688 >> 172.21.15.173:0/2809715688 conn(0x7f3aac0f0b70 msgr2=0x7f3aac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:15.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.000+0000 7f3ab1765700 1 -- 172.21.15.173:0/2809715688 shutdown_connections 2024-01-21T19:56:15.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.000+0000 7f3ab1765700 1 -- 172.21.15.173:0/2809715688 wait complete. 2024-01-21T19:56:15.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.001+0000 7f3ab1765700 1 Processor -- start 2024-01-21T19:56:15.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.001+0000 7f3ab1765700 1 -- start start 2024-01-21T19:56:15.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.001+0000 7f3ab1765700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3aac0f5410 0x7f3aac105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:15.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.001+0000 7f3ab1765700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac105af0 con 0x7f3aac0f5410 2024-01-21T19:56:15.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.001+0000 7f3aaaffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3aac0f5410 0x7f3aac105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:15.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.001+0000 7f3aaaffd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3aac0f5410 0x7f3aac105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:36096/0 (socket says 172.21.15.173:36096) 2024-01-21T19:56:15.998 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.001+0000 7f3aaaffd700 1 -- 172.21.15.173:0/325614864 learned_addr learned my addr 172.21.15.173:0/325614864 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:56:15.998 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.002+0000 7f3aaaffd700 1 -- 172.21.15.173:0/325614864 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a94005040 con 0x7f3aac0f5410 2024-01-21T19:56:15.998 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.002+0000 7f3aaaffd700 1 --2- 172.21.15.173:0/325614864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3aac0f5410 0x7f3aac105600 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f3a94013510 tx=0x7f3a9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:15.998 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.002+0000 7f3aa37fe700 1 -- 172.21.15.173:0/325614864 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a94009320 con 0x7f3aac0f5410 2024-01-21T19:56:15.998 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.002+0000 7f3aa37fe700 1 -- 172.21.15.173:0/325614864 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a94013f00 con 0x7f3aac0f5410 2024-01-21T19:56:15.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.002+0000 7f3ab1765700 1 -- 172.21.15.173:0/325614864 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aac105cf0 con 0x7f3aac0f5410 2024-01-21T19:56:15.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.002+0000 7f3aa37fe700 1 -- 172.21.15.173:0/325614864 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a94014d90 con 0x7f3aac0f5410 2024-01-21T19:56:15.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.002+0000 7f3ab1765700 1 -- 172.21.15.173:0/325614864 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aac106180 con 0x7f3aac0f5410 2024-01-21T19:56:15.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.003+0000 7f3aa37fe700 1 -- 172.21.15.173:0/325614864 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3a940263a0 con 0x7f3aac0f5410 2024-01-21T19:56:15.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.003+0000 7f3ab1765700 1 -- 172.21.15.173:0/325614864 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aac045ad0 con 0x7f3aac0f5410 2024-01-21T19:56:15.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.003+0000 7f3aa37fe700 1 --2- 172.21.15.173:0/325614864 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3a9802ffb0 0x7f3a98032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:16.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.003+0000 7f3aa37fe700 1 -- 172.21.15.173:0/325614864 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3a94041360 con 0x7f3aac0f5410 2024-01-21T19:56:16.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.003+0000 7f3aaa7fc700 1 --2- 172.21.15.173:0/325614864 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3a9802ffb0 0x7f3a98032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:16.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.004+0000 7f3aaa7fc700 1 --2- 172.21.15.173:0/325614864 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3a9802ffb0 0x7f3a98032470 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f3a9c0096a0 tx=0x7f3a9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:16.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.008+0000 7f3aa37fe700 1 -- 172.21.15.173:0/325614864 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a940126f0 con 0x7f3aac0f5410 2024-01-21T19:56:16.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.212+0000 7f3ab1765700 1 -- 172.21.15.173:0/325614864 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3aac0fae60 con 0x7f3aac0f5410 2024-01-21T19:56:16.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.213+0000 7f3aa37fe700 1 -- 172.21.15.173:0/325614864 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3a94012330 con 0x7f3aac0f5410 2024-01-21T19:56:16.204 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:56:16.205 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:56:16.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.215+0000 7f3ab1765700 1 -- 172.21.15.173:0/325614864 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3a9802ffb0 msgr2=0x7f3a98032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:16.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.215+0000 7f3ab1765700 1 --2- 172.21.15.173:0/325614864 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3a9802ffb0 0x7f3a98032470 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f3a9c0096a0 tx=0x7f3a9c006b40).stop 2024-01-21T19:56:16.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.215+0000 7f3ab1765700 1 -- 172.21.15.173:0/325614864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3aac0f5410 msgr2=0x7f3aac105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:16.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.215+0000 7f3ab1765700 1 --2- 172.21.15.173:0/325614864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3aac0f5410 0x7f3aac105600 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f3a94013510 tx=0x7f3a9400b040).stop 2024-01-21T19:56:16.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.215+0000 7f3ab1765700 1 -- 172.21.15.173:0/325614864 shutdown_connections 2024-01-21T19:56:16.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.215+0000 7f3ab1765700 1 --2- 172.21.15.173:0/325614864 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3a9802ffb0 0x7f3a98032470 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:16.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.215+0000 7f3ab1765700 1 --2- 172.21.15.173:0/325614864 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3aac0f5410 0x7f3aac105600 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:16.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.215+0000 7f3ab1765700 1 -- 172.21.15.173:0/325614864 >> 172.21.15.173:0/325614864 conn(0x7f3aac0f0b70 msgr2=0x7f3aac0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:16.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.215+0000 7f3ab1765700 1 -- 172.21.15.173:0/325614864 shutdown_connections 2024-01-21T19:56:16.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:16.215+0000 7f3ab1765700 1 -- 172.21.15.173:0/325614864 wait complete. 2024-01-21T19:56:16.208 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:17.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:16 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:15.007493+0000 mgr.smithi169.espgue (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:17.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:16 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:16.213891+0000 mon.smithi169 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.173:0/325614864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:17.595 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:17.595 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:56:18.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.827+0000 7fa902702700 1 -- 172.21.15.173:0/229714928 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8fc0f6510 msgr2=0x7fa8fc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:18.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.827+0000 7fa902702700 1 --2- 172.21.15.173:0/229714928 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8fc0f6510 0x7fa8fc0f6930 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fa8ec009fb0 tx=0x7fa8ec009b30).stop 2024-01-21T19:56:18.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.827+0000 7fa902702700 1 -- 172.21.15.173:0/229714928 shutdown_connections 2024-01-21T19:56:18.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.828+0000 7fa902702700 1 --2- 172.21.15.173:0/229714928 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8fc0f6510 0x7fa8fc0f6930 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:18.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.828+0000 7fa902702700 1 -- 172.21.15.173:0/229714928 >> 172.21.15.173:0/229714928 conn(0x7fa8fc0f1c70 msgr2=0x7fa8fc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:18.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.828+0000 7fa902702700 1 -- 172.21.15.173:0/229714928 shutdown_connections 2024-01-21T19:56:18.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.828+0000 7fa902702700 1 -- 172.21.15.173:0/229714928 wait complete. 2024-01-21T19:56:18.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.828+0000 7fa902702700 1 Processor -- start 2024-01-21T19:56:18.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.828+0000 7fa902702700 1 -- start start 2024-01-21T19:56:18.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.828+0000 7fa902702700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8fc0f6510 0x7fa8fc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:18.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.828+0000 7fa902702700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8fc069e20 con 0x7fa8fc0f6510 2024-01-21T19:56:18.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.829+0000 7fa8fbfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8fc0f6510 0x7fa8fc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:18.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.829+0000 7fa8fbfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8fc0f6510 0x7fa8fc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:36122/0 (socket says 172.21.15.173:36122) 2024-01-21T19:56:18.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.829+0000 7fa8fbfff700 1 -- 172.21.15.173:0/1301320008 learned_addr learned my addr 172.21.15.173:0/1301320008 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:56:18.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.829+0000 7fa8fbfff700 1 -- 172.21.15.173:0/1301320008 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ec005040 con 0x7fa8fc0f6510 2024-01-21T19:56:18.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.829+0000 7fa8fbfff700 1 --2- 172.21.15.173:0/1301320008 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8fc0f6510 0x7fa8fc06b9a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fa8ec013510 tx=0x7fa8ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:18.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.829+0000 7fa8f8ff9700 1 -- 172.21.15.173:0/1301320008 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8ec009320 con 0x7fa8fc0f6510 2024-01-21T19:56:18.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.829+0000 7fa8f8ff9700 1 -- 172.21.15.173:0/1301320008 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8ec013f00 con 0x7fa8fc0f6510 2024-01-21T19:56:18.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.829+0000 7fa902702700 1 -- 172.21.15.173:0/1301320008 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8fc069fc0 con 0x7fa8fc0f6510 2024-01-21T19:56:18.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.829+0000 7fa8f8ff9700 1 -- 172.21.15.173:0/1301320008 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8ec014d90 con 0x7fa8fc0f6510 2024-01-21T19:56:18.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.829+0000 7fa902702700 1 -- 172.21.15.173:0/1301320008 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8fc06a450 con 0x7fa8fc0f6510 2024-01-21T19:56:18.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.830+0000 7fa8f8ff9700 1 -- 172.21.15.173:0/1301320008 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8ec014420 con 0x7fa8fc0f6510 2024-01-21T19:56:18.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.830+0000 7fa8f8ff9700 1 --2- 172.21.15.173:0/1301320008 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa8e402ff60 0x7fa8e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:18.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.830+0000 7fa902702700 1 -- 172.21.15.173:0/1301320008 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8fc059100 con 0x7fa8fc0f6510 2024-01-21T19:56:18.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.830+0000 7fa8f8ff9700 1 -- 172.21.15.173:0/1301320008 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8ec041110 con 0x7fa8fc0f6510 2024-01-21T19:56:18.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.830+0000 7fa8fb7fe700 1 --2- 172.21.15.173:0/1301320008 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa8e402ff60 0x7fa8e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:18.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.831+0000 7fa8fb7fe700 1 --2- 172.21.15.173:0/1301320008 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa8e402ff60 0x7fa8e4032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fa8f00096a0 tx=0x7fa8f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:18.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:18.835+0000 7fa8f8ff9700 1 -- 172.21.15.173:0/1301320008 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8ec012330 con 0x7fa8fc0f6510 2024-01-21T19:56:19.029 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:19.038+0000 7fa902702700 1 -- 172.21.15.173:0/1301320008 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8fc045ad0 con 0x7fa8fc0f6510 2024-01-21T19:56:19.029 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:19.038+0000 7fa8f8ff9700 1 -- 172.21.15.173:0/1301320008 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa8ec01db70 con 0x7fa8fc0f6510 2024-01-21T19:56:19.030 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:56:19.030 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:56:19.031 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:19.040+0000 7fa902702700 1 -- 172.21.15.173:0/1301320008 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa8e402ff60 msgr2=0x7fa8e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:19.031 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:19.040+0000 7fa902702700 1 --2- 172.21.15.173:0/1301320008 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa8e402ff60 0x7fa8e4032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fa8f00096a0 tx=0x7fa8f0006b40).stop 2024-01-21T19:56:19.031 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:19.040+0000 7fa902702700 1 -- 172.21.15.173:0/1301320008 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8fc0f6510 msgr2=0x7fa8fc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:19.031 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:19.040+0000 7fa902702700 1 --2- 172.21.15.173:0/1301320008 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8fc0f6510 0x7fa8fc06b9a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fa8ec013510 tx=0x7fa8ec00b040).stop 2024-01-21T19:56:19.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:19.040+0000 7fa902702700 1 -- 172.21.15.173:0/1301320008 shutdown_connections 2024-01-21T19:56:19.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:19.040+0000 7fa902702700 1 --2- 172.21.15.173:0/1301320008 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa8e402ff60 0x7fa8e4032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:19.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:19.040+0000 7fa902702700 1 --2- 172.21.15.173:0/1301320008 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8fc0f6510 0x7fa8fc06b9a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:19.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:19.040+0000 7fa902702700 1 -- 172.21.15.173:0/1301320008 >> 172.21.15.173:0/1301320008 conn(0x7fa8fc0f1c70 msgr2=0x7fa8fc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:19.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:19.040+0000 7fa902702700 1 -- 172.21.15.173:0/1301320008 shutdown_connections 2024-01-21T19:56:19.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:19.040+0000 7fa902702700 1 -- 172.21.15.173:0/1301320008 wait complete. 2024-01-21T19:56:19.032 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:19.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:18 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:17.007869+0000 mgr.smithi169.espgue (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:20.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:19 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:19.039020+0000 mon.smithi169 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.173:0/1301320008' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:20.419 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:20.420 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:56:21.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:20 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:19.008160+0000 mgr.smithi169.espgue (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:21.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.655+0000 7ffabb896700 1 -- 172.21.15.173:0/120807386 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffab40f5410 msgr2=0x7ffab40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:21.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.655+0000 7ffabb896700 1 --2- 172.21.15.173:0/120807386 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffab40f5410 0x7ffab40f5830 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7ffab0009fb0 tx=0x7ffab0009b30).stop 2024-01-21T19:56:21.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.655+0000 7ffabb896700 1 -- 172.21.15.173:0/120807386 shutdown_connections 2024-01-21T19:56:21.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.655+0000 7ffabb896700 1 --2- 172.21.15.173:0/120807386 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffab40f5410 0x7ffab40f5830 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:21.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.655+0000 7ffabb896700 1 -- 172.21.15.173:0/120807386 >> 172.21.15.173:0/120807386 conn(0x7ffab40f0b70 msgr2=0x7ffab40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:21.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.655+0000 7ffabb896700 1 -- 172.21.15.173:0/120807386 shutdown_connections 2024-01-21T19:56:21.648 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.655+0000 7ffabb896700 1 -- 172.21.15.173:0/120807386 wait complete. 2024-01-21T19:56:21.651 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.656+0000 7ffabb896700 1 Processor -- start 2024-01-21T19:56:21.651 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.656+0000 7ffabb896700 1 -- start start 2024-01-21T19:56:21.651 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.656+0000 7ffabb896700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffab40f5410 0x7ffab4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:21.651 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.656+0000 7ffabb896700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab4105b20 con 0x7ffab40f5410 2024-01-21T19:56:21.651 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.656+0000 7ffab9632700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffab40f5410 0x7ffab4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:21.651 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.657+0000 7ffab9632700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffab40f5410 0x7ffab4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:36138/0 (socket says 172.21.15.173:36138) 2024-01-21T19:56:21.651 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.657+0000 7ffab9632700 1 -- 172.21.15.173:0/2718749793 learned_addr learned my addr 172.21.15.173:0/2718749793 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:56:21.651 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.657+0000 7ffab9632700 1 -- 172.21.15.173:0/2718749793 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab0005040 con 0x7ffab40f5410 2024-01-21T19:56:21.651 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.657+0000 7ffab9632700 1 --2- 172.21.15.173:0/2718749793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffab40f5410 0x7ffab4105630 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7ffab0013510 tx=0x7ffab000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:21.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.657+0000 7ffaa9ffb700 1 -- 172.21.15.173:0/2718749793 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffab0009320 con 0x7ffab40f5410 2024-01-21T19:56:21.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.657+0000 7ffabb896700 1 -- 172.21.15.173:0/2718749793 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab4105d20 con 0x7ffab40f5410 2024-01-21T19:56:21.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.657+0000 7ffaa9ffb700 1 -- 172.21.15.173:0/2718749793 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffab0013f00 con 0x7ffab40f5410 2024-01-21T19:56:21.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.657+0000 7ffaa9ffb700 1 -- 172.21.15.173:0/2718749793 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffab0014d90 con 0x7ffab40f5410 2024-01-21T19:56:21.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.657+0000 7ffabb896700 1 -- 172.21.15.173:0/2718749793 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab41061b0 con 0x7ffab40f5410 2024-01-21T19:56:21.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.658+0000 7ffaa9ffb700 1 -- 172.21.15.173:0/2718749793 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffab00095f0 con 0x7ffab40f5410 2024-01-21T19:56:21.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.658+0000 7ffabb896700 1 -- 172.21.15.173:0/2718749793 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffab4045ad0 con 0x7ffab40f5410 2024-01-21T19:56:21.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.658+0000 7ffaa9ffb700 1 --2- 172.21.15.173:0/2718749793 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ffaa002ff60 0x7ffaa0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:21.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.658+0000 7ffaa9ffb700 1 -- 172.21.15.173:0/2718749793 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffab00410c0 con 0x7ffab40f5410 2024-01-21T19:56:21.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.659+0000 7ffab8e31700 1 --2- 172.21.15.173:0/2718749793 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ffaa002ff60 0x7ffaa0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:21.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.659+0000 7ffab8e31700 1 --2- 172.21.15.173:0/2718749793 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ffaa002ff60 0x7ffaa0032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7ffaa40096a0 tx=0x7ffaa4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:21.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.663+0000 7ffaa9ffb700 1 -- 172.21.15.173:0/2718749793 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffab001d7f0 con 0x7ffab40f5410 2024-01-21T19:56:21.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.868+0000 7ffabb896700 1 -- 172.21.15.173:0/2718749793 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffab40fae60 con 0x7ffab40f5410 2024-01-21T19:56:21.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.869+0000 7ffaa9ffb700 1 -- 172.21.15.173:0/2718749793 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffab0010230 con 0x7ffab40f5410 2024-01-21T19:56:21.860 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:56:21.860 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:56:21.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.870+0000 7ffabb896700 1 -- 172.21.15.173:0/2718749793 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ffaa002ff60 msgr2=0x7ffaa0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:21.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.870+0000 7ffabb896700 1 --2- 172.21.15.173:0/2718749793 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ffaa002ff60 0x7ffaa0032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7ffaa40096a0 tx=0x7ffaa4006b40).stop 2024-01-21T19:56:21.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.870+0000 7ffabb896700 1 -- 172.21.15.173:0/2718749793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffab40f5410 msgr2=0x7ffab4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:21.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.870+0000 7ffabb896700 1 --2- 172.21.15.173:0/2718749793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffab40f5410 0x7ffab4105630 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7ffab0013510 tx=0x7ffab000b040).stop 2024-01-21T19:56:21.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.870+0000 7ffabb896700 1 -- 172.21.15.173:0/2718749793 shutdown_connections 2024-01-21T19:56:21.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.870+0000 7ffabb896700 1 --2- 172.21.15.173:0/2718749793 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7ffaa002ff60 0x7ffaa0032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:21.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.870+0000 7ffabb896700 1 --2- 172.21.15.173:0/2718749793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffab40f5410 0x7ffab4105630 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:21.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.870+0000 7ffabb896700 1 -- 172.21.15.173:0/2718749793 >> 172.21.15.173:0/2718749793 conn(0x7ffab40f0b70 msgr2=0x7ffab40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:21.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.871+0000 7ffabb896700 1 -- 172.21.15.173:0/2718749793 shutdown_connections 2024-01-21T19:56:21.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:21.871+0000 7ffabb896700 1 -- 172.21.15.173:0/2718749793 wait complete. 2024-01-21T19:56:21.863 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:23.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:22 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:21.008441+0000 mgr.smithi169.espgue (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:23.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:22 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:21.869604+0000 mon.smithi169 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.173:0/2718749793' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:23.253 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:23.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:56:24.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.455+0000 7f32b6ffd700 1 -- 172.21.15.173:0/1811350325 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32a80138a0 con 0x7f32b80f5410 2024-01-21T19:56:24.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.455+0000 7f32be94f700 1 -- 172.21.15.173:0/1811350325 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32b80f5410 msgr2=0x7f32b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:24.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.455+0000 7f32be94f700 1 --2- 172.21.15.173:0/1811350325 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32b80f5410 0x7f32b80f5830 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f32a8009fb0 tx=0x7f32a8009b30).stop 2024-01-21T19:56:24.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.455+0000 7f32be94f700 1 -- 172.21.15.173:0/1811350325 shutdown_connections 2024-01-21T19:56:24.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.455+0000 7f32be94f700 1 --2- 172.21.15.173:0/1811350325 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32b80f5410 0x7f32b80f5830 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:24.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.455+0000 7f32be94f700 1 -- 172.21.15.173:0/1811350325 >> 172.21.15.173:0/1811350325 conn(0x7f32b80f0b70 msgr2=0x7f32b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:24.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.455+0000 7f32be94f700 1 -- 172.21.15.173:0/1811350325 shutdown_connections 2024-01-21T19:56:24.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.455+0000 7f32be94f700 1 -- 172.21.15.173:0/1811350325 wait complete. 2024-01-21T19:56:24.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.456+0000 7f32be94f700 1 Processor -- start 2024-01-21T19:56:24.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.456+0000 7f32be94f700 1 -- start start 2024-01-21T19:56:24.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.456+0000 7f32be94f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32b80f5410 0x7f32b8108930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:24.449 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.456+0000 7f32be94f700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b8108e20 con 0x7f32b80f5410 2024-01-21T19:56:24.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.456+0000 7f32b7fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32b80f5410 0x7f32b8108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:24.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.457+0000 7f32b7fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32b80f5410 0x7f32b8108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:59136/0 (socket says 172.21.15.173:59136) 2024-01-21T19:56:24.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.457+0000 7f32b7fff700 1 -- 172.21.15.173:0/3522873692 learned_addr learned my addr 172.21.15.173:0/3522873692 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:56:24.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.457+0000 7f32b7fff700 1 -- 172.21.15.173:0/3522873692 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a8005040 con 0x7f32b80f5410 2024-01-21T19:56:24.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.457+0000 7f32b7fff700 1 --2- 172.21.15.173:0/3522873692 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32b80f5410 0x7f32b8108930 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f32a8013510 tx=0x7f32a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:24.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.457+0000 7f32b4ff9700 1 -- 172.21.15.173:0/3522873692 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32a8009320 con 0x7f32b80f5410 2024-01-21T19:56:24.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.457+0000 7f32be94f700 1 -- 172.21.15.173:0/3522873692 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b8109020 con 0x7f32b80f5410 2024-01-21T19:56:24.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.457+0000 7f32b4ff9700 1 -- 172.21.15.173:0/3522873692 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32a8013f00 con 0x7f32b80f5410 2024-01-21T19:56:24.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.457+0000 7f32b4ff9700 1 -- 172.21.15.173:0/3522873692 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32a8014d90 con 0x7f32b80f5410 2024-01-21T19:56:24.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.457+0000 7f32be94f700 1 -- 172.21.15.173:0/3522873692 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b81094b0 con 0x7f32b80f5410 2024-01-21T19:56:24.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.458+0000 7f32b4ff9700 1 -- 172.21.15.173:0/3522873692 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f32a80263a0 con 0x7f32b80f5410 2024-01-21T19:56:24.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.458+0000 7f32be94f700 1 -- 172.21.15.173:0/3522873692 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b8045ad0 con 0x7f32b80f5410 2024-01-21T19:56:24.452 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.458+0000 7f32b4ff9700 1 --2- 172.21.15.173:0/3522873692 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f32a002ffb0 0x7f32a0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:24.452 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.458+0000 7f32b4ff9700 1 -- 172.21.15.173:0/3522873692 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f32a80413c0 con 0x7f32b80f5410 2024-01-21T19:56:24.452 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.458+0000 7f32b77fe700 1 --2- 172.21.15.173:0/3522873692 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f32a002ffb0 0x7f32a0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:24.452 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.459+0000 7f32b77fe700 1 --2- 172.21.15.173:0/3522873692 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f32a002ffb0 0x7f32a0032470 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f32ac0096a0 tx=0x7f32ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:24.454 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.463+0000 7f32b4ff9700 1 -- 172.21.15.173:0/3522873692 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32a80126f0 con 0x7f32b80f5410 2024-01-21T19:56:24.659 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.668+0000 7f32be94f700 1 -- 172.21.15.173:0/3522873692 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f32b80fe190 con 0x7f32b80f5410 2024-01-21T19:56:24.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.668+0000 7f32b4ff9700 1 -- 172.21.15.173:0/3522873692 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f32a800f340 con 0x7f32b80f5410 2024-01-21T19:56:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:56:24.660 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:56:24.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.670+0000 7f32be94f700 1 -- 172.21.15.173:0/3522873692 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f32a002ffb0 msgr2=0x7f32a0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:24.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.670+0000 7f32be94f700 1 --2- 172.21.15.173:0/3522873692 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f32a002ffb0 0x7f32a0032470 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f32ac0096a0 tx=0x7f32ac006b40).stop 2024-01-21T19:56:24.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.670+0000 7f32be94f700 1 -- 172.21.15.173:0/3522873692 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32b80f5410 msgr2=0x7f32b8108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:24.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.670+0000 7f32be94f700 1 --2- 172.21.15.173:0/3522873692 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32b80f5410 0x7f32b8108930 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f32a8013510 tx=0x7f32a800b040).stop 2024-01-21T19:56:24.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.670+0000 7f32be94f700 1 -- 172.21.15.173:0/3522873692 shutdown_connections 2024-01-21T19:56:24.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.670+0000 7f32be94f700 1 --2- 172.21.15.173:0/3522873692 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f32a002ffb0 0x7f32a0032470 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:24.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.670+0000 7f32be94f700 1 --2- 172.21.15.173:0/3522873692 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f32b80f5410 0x7f32b8108930 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:24.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.670+0000 7f32be94f700 1 -- 172.21.15.173:0/3522873692 >> 172.21.15.173:0/3522873692 conn(0x7f32b80f0b70 msgr2=0x7f32b80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:24.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.670+0000 7f32be94f700 1 -- 172.21.15.173:0/3522873692 shutdown_connections 2024-01-21T19:56:24.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:24.670+0000 7f32be94f700 1 -- 172.21.15.173:0/3522873692 wait complete. 2024-01-21T19:56:24.662 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:25.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:24 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:23.008757+0000 mgr.smithi169.espgue (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:25.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:24 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:24.669112+0000 mon.smithi169 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.173:0/3522873692' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:26.066 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:26.066 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:56:27.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:26 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:25.009092+0000 mgr.smithi169.espgue (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:27.241 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.249+0000 7f0c3b59e700 1 -- 172.21.15.173:0/4050426523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c3c0f62a0 msgr2=0x7f0c3c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:27.242 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.249+0000 7f0c3b59e700 1 --2- 172.21.15.173:0/4050426523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c3c0f62a0 0x7f0c3c0f66c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f0c30009fb0 tx=0x7f0c30009b30).stop 2024-01-21T19:56:27.242 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.249+0000 7f0c3b59e700 1 -- 172.21.15.173:0/4050426523 shutdown_connections 2024-01-21T19:56:27.242 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.249+0000 7f0c3b59e700 1 --2- 172.21.15.173:0/4050426523 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c3c0f62a0 0x7f0c3c0f66c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:27.242 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.249+0000 7f0c3b59e700 1 -- 172.21.15.173:0/4050426523 >> 172.21.15.173:0/4050426523 conn(0x7f0c3c0f1ac0 msgr2=0x7f0c3c0f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:27.242 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.249+0000 7f0c3b59e700 1 -- 172.21.15.173:0/4050426523 shutdown_connections 2024-01-21T19:56:27.242 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.249+0000 7f0c3b59e700 1 -- 172.21.15.173:0/4050426523 wait complete. 2024-01-21T19:56:27.242 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.249+0000 7f0c3b59e700 1 Processor -- start 2024-01-21T19:56:27.242 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.249+0000 7f0c3b59e700 1 -- start start 2024-01-21T19:56:27.242 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.249+0000 7f0c3b59e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c3c0f62a0 0x7f0c3c1064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:27.242 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.249+0000 7f0c3b59e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c106990 con 0x7f0c3c0f62a0 2024-01-21T19:56:27.242 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.250+0000 7f0c3a59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c3c0f62a0 0x7f0c3c1064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:27.242 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.250+0000 7f0c3a59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c3c0f62a0 0x7f0c3c1064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:59150/0 (socket says 172.21.15.173:59150) 2024-01-21T19:56:27.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.250+0000 7f0c3a59c700 1 -- 172.21.15.173:0/2343617623 learned_addr learned my addr 172.21.15.173:0/2343617623 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:56:27.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.250+0000 7f0c3a59c700 1 -- 172.21.15.173:0/2343617623 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c30005040 con 0x7f0c3c0f62a0 2024-01-21T19:56:27.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.250+0000 7f0c3a59c700 1 --2- 172.21.15.173:0/2343617623 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c3c0f62a0 0x7f0c3c1064a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f0c30013510 tx=0x7f0c3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:27.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.250+0000 7f0c2affd700 1 -- 172.21.15.173:0/2343617623 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c30009320 con 0x7f0c3c0f62a0 2024-01-21T19:56:27.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.250+0000 7f0c3b59e700 1 -- 172.21.15.173:0/2343617623 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c3c106b90 con 0x7f0c3c0f62a0 2024-01-21T19:56:27.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.250+0000 7f0c2affd700 1 -- 172.21.15.173:0/2343617623 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c30013f00 con 0x7f0c3c0f62a0 2024-01-21T19:56:27.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.250+0000 7f0c2affd700 1 -- 172.21.15.173:0/2343617623 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c30014dc0 con 0x7f0c3c0f62a0 2024-01-21T19:56:27.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.250+0000 7f0c3b59e700 1 -- 172.21.15.173:0/2343617623 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c3c107020 con 0x7f0c3c0f62a0 2024-01-21T19:56:27.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.251+0000 7f0c2affd700 1 -- 172.21.15.173:0/2343617623 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c300095f0 con 0x7f0c3c0f62a0 2024-01-21T19:56:27.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.251+0000 7f0c3b59e700 1 -- 172.21.15.173:0/2343617623 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c3c045ad0 con 0x7f0c3c0f62a0 2024-01-21T19:56:27.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.251+0000 7f0c2affd700 1 --2- 172.21.15.173:0/2343617623 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0c2402ff60 0x7f0c24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:27.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.251+0000 7f0c2affd700 1 -- 172.21.15.173:0/2343617623 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c300410c0 con 0x7f0c3c0f62a0 2024-01-21T19:56:27.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.252+0000 7f0c39d9b700 1 --2- 172.21.15.173:0/2343617623 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0c2402ff60 0x7f0c24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:27.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.252+0000 7f0c39d9b700 1 --2- 172.21.15.173:0/2343617623 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0c2402ff60 0x7f0c24032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f0c2c0096a0 tx=0x7f0c2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:27.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.256+0000 7f0c2affd700 1 -- 172.21.15.173:0/2343617623 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c3001c790 con 0x7f0c3c0f62a0 2024-01-21T19:56:27.457 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.466+0000 7f0c3b59e700 1 -- 172.21.15.173:0/2343617623 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c3c107ed0 con 0x7f0c3c0f62a0 2024-01-21T19:56:27.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.467+0000 7f0c2affd700 1 -- 172.21.15.173:0/2343617623 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0c30010230 con 0x7f0c3c0f62a0 2024-01-21T19:56:27.458 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:56:27.458 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:56:27.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.469+0000 7f0c3b59e700 1 -- 172.21.15.173:0/2343617623 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0c2402ff60 msgr2=0x7f0c24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:27.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.469+0000 7f0c3b59e700 1 --2- 172.21.15.173:0/2343617623 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0c2402ff60 0x7f0c24032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f0c2c0096a0 tx=0x7f0c2c006b40).stop 2024-01-21T19:56:27.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.469+0000 7f0c3b59e700 1 -- 172.21.15.173:0/2343617623 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c3c0f62a0 msgr2=0x7f0c3c1064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:27.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.469+0000 7f0c3b59e700 1 --2- 172.21.15.173:0/2343617623 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c3c0f62a0 0x7f0c3c1064a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f0c30013510 tx=0x7f0c3000b040).stop 2024-01-21T19:56:27.461 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.469+0000 7f0c3b59e700 1 -- 172.21.15.173:0/2343617623 shutdown_connections 2024-01-21T19:56:27.461 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.469+0000 7f0c3b59e700 1 --2- 172.21.15.173:0/2343617623 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f0c2402ff60 0x7f0c24032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:27.461 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.469+0000 7f0c3b59e700 1 --2- 172.21.15.173:0/2343617623 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0c3c0f62a0 0x7f0c3c1064a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:27.461 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.469+0000 7f0c3b59e700 1 -- 172.21.15.173:0/2343617623 >> 172.21.15.173:0/2343617623 conn(0x7f0c3c0f1ac0 msgr2=0x7f0c3c0f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:27.461 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.469+0000 7f0c3b59e700 1 -- 172.21.15.173:0/2343617623 shutdown_connections 2024-01-21T19:56:27.461 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:27.469+0000 7f0c3b59e700 1 -- 172.21.15.173:0/2343617623 wait complete. 2024-01-21T19:56:27.461 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:28.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:27 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:27.467738+0000 mon.smithi169 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.173:0/2343617623' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:28.855 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:28.855 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:56:29.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:28 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:27.009356+0000 mgr.smithi169.espgue (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:30.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.039+0000 7f75929d4700 1 -- 172.21.15.173:0/4289455637 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f758c0f51a0 msgr2=0x7f758c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:30.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.039+0000 7f75929d4700 1 --2- 172.21.15.173:0/4289455637 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f758c0f51a0 0x7f758c0f55c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f757c009fb0 tx=0x7f757c009b30).stop 2024-01-21T19:56:30.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.039+0000 7f75929d4700 1 -- 172.21.15.173:0/4289455637 shutdown_connections 2024-01-21T19:56:30.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.039+0000 7f75929d4700 1 --2- 172.21.15.173:0/4289455637 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f758c0f51a0 0x7f758c0f55c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:30.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.039+0000 7f75929d4700 1 -- 172.21.15.173:0/4289455637 >> 172.21.15.173:0/4289455637 conn(0x7f758c0f09a0 msgr2=0x7f758c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:30.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.040+0000 7f75929d4700 1 -- 172.21.15.173:0/4289455637 shutdown_connections 2024-01-21T19:56:30.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.040+0000 7f75929d4700 1 -- 172.21.15.173:0/4289455637 wait complete. 2024-01-21T19:56:30.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.040+0000 7f75929d4700 1 Processor -- start 2024-01-21T19:56:30.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.040+0000 7f75929d4700 1 -- start start 2024-01-21T19:56:30.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.040+0000 7f75929d4700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f758c0f51a0 0x7f758c106530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:30.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.040+0000 7f75929d4700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758c106a20 con 0x7f758c0f51a0 2024-01-21T19:56:30.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.041+0000 7f75919d2700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f758c0f51a0 0x7f758c106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:30.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.041+0000 7f75919d2700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f758c0f51a0 0x7f758c106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:59166/0 (socket says 172.21.15.173:59166) 2024-01-21T19:56:30.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.041+0000 7f75919d2700 1 -- 172.21.15.173:0/2941774558 learned_addr learned my addr 172.21.15.173:0/2941774558 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:56:30.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.041+0000 7f75919d2700 1 -- 172.21.15.173:0/2941774558 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c005040 con 0x7f758c0f51a0 2024-01-21T19:56:30.035 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.041+0000 7f75919d2700 1 --2- 172.21.15.173:0/2941774558 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f758c0f51a0 0x7f758c106530 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f757c009f80 tx=0x7f757c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:30.035 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.041+0000 7f75827fc700 1 -- 172.21.15.173:0/2941774558 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f757c009320 con 0x7f758c0f51a0 2024-01-21T19:56:30.035 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.041+0000 7f75827fc700 1 -- 172.21.15.173:0/2941774558 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f757c013ec0 con 0x7f758c0f51a0 2024-01-21T19:56:30.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.041+0000 7f75929d4700 1 -- 172.21.15.173:0/2941774558 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f758c106c20 con 0x7f758c0f51a0 2024-01-21T19:56:30.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.041+0000 7f75827fc700 1 -- 172.21.15.173:0/2941774558 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f757c01ec80 con 0x7f758c0f51a0 2024-01-21T19:56:30.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.041+0000 7f75929d4700 1 -- 172.21.15.173:0/2941774558 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f758c1070b0 con 0x7f758c0f51a0 2024-01-21T19:56:30.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.042+0000 7f75827fc700 1 -- 172.21.15.173:0/2941774558 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f757c01e420 con 0x7f758c0f51a0 2024-01-21T19:56:30.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.042+0000 7f75929d4700 1 -- 172.21.15.173:0/2941774558 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f758c045ad0 con 0x7f758c0f51a0 2024-01-21T19:56:30.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.042+0000 7f75827fc700 1 --2- 172.21.15.173:0/2941774558 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f757802ff60 0x7f7578032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:30.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.042+0000 7f75827fc700 1 -- 172.21.15.173:0/2941774558 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f757c040e60 con 0x7f758c0f51a0 2024-01-21T19:56:30.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.043+0000 7f75911d1700 1 --2- 172.21.15.173:0/2941774558 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f757802ff60 0x7f7578032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:30.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.043+0000 7f75911d1700 1 --2- 172.21.15.173:0/2941774558 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f757802ff60 0x7f7578032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f75880096a0 tx=0x7f7588006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:30.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.047+0000 7f75827fc700 1 -- 172.21.15.173:0/2941774558 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f757c013380 con 0x7f758c0f51a0 2024-01-21T19:56:30.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.252+0000 7f75929d4700 1 -- 172.21.15.173:0/2941774558 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f758c107f00 con 0x7f758c0f51a0 2024-01-21T19:56:30.245 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.253+0000 7f75827fc700 1 -- 172.21.15.173:0/2941774558 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f757c02c2d0 con 0x7f758c0f51a0 2024-01-21T19:56:30.245 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:56:30.245 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:56:30.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.254+0000 7f75929d4700 1 -- 172.21.15.173:0/2941774558 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f757802ff60 msgr2=0x7f7578032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:30.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.254+0000 7f75929d4700 1 --2- 172.21.15.173:0/2941774558 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f757802ff60 0x7f7578032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f75880096a0 tx=0x7f7588006b40).stop 2024-01-21T19:56:30.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.254+0000 7f75929d4700 1 -- 172.21.15.173:0/2941774558 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f758c0f51a0 msgr2=0x7f758c106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:30.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.254+0000 7f75929d4700 1 --2- 172.21.15.173:0/2941774558 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f758c0f51a0 0x7f758c106530 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f757c009f80 tx=0x7f757c00b040).stop 2024-01-21T19:56:30.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.254+0000 7f75929d4700 1 -- 172.21.15.173:0/2941774558 shutdown_connections 2024-01-21T19:56:30.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.254+0000 7f75929d4700 1 --2- 172.21.15.173:0/2941774558 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f757802ff60 0x7f7578032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:30.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.254+0000 7f75929d4700 1 --2- 172.21.15.173:0/2941774558 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f758c0f51a0 0x7f758c106530 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:30.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.254+0000 7f75929d4700 1 -- 172.21.15.173:0/2941774558 >> 172.21.15.173:0/2941774558 conn(0x7f758c0f09a0 msgr2=0x7f758c05fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:30.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.254+0000 7f75929d4700 1 -- 172.21.15.173:0/2941774558 shutdown_connections 2024-01-21T19:56:30.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:30.254+0000 7f75929d4700 1 -- 172.21.15.173:0/2941774558 wait complete. 2024-01-21T19:56:30.248 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:31.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:30 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:29.009731+0000 mgr.smithi169.espgue (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:31.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:30 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:30.254315+0000 mon.smithi169 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.173:0/2941774558' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:31.648 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:31.648 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:56:32.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:31.077022+0000 mon.smithi169 (mon.0) 300 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/mirror_snapshot_schedule"}]: dispatch 2024-01-21T19:56:32.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:31.080079+0000 mon.smithi169 (mon.0) 301 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/trash_purge_schedule"}]: dispatch 2024-01-21T19:56:32.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.852+0000 7fa60633f700 1 -- 172.21.15.173:0/969829349 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6000f51a0 msgr2=0x7fa6000f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:32.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.852+0000 7fa60633f700 1 --2- 172.21.15.173:0/969829349 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6000f51a0 0x7fa6000f55c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fa5fc009fb0 tx=0x7fa5fc009b30).stop 2024-01-21T19:56:32.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.852+0000 7fa60633f700 1 -- 172.21.15.173:0/969829349 shutdown_connections 2024-01-21T19:56:32.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.852+0000 7fa60633f700 1 --2- 172.21.15.173:0/969829349 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6000f51a0 0x7fa6000f55c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:32.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.852+0000 7fa60633f700 1 -- 172.21.15.173:0/969829349 >> 172.21.15.173:0/969829349 conn(0x7fa6000f09a0 msgr2=0x7fa6000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:32.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.852+0000 7fa60633f700 1 -- 172.21.15.173:0/969829349 shutdown_connections 2024-01-21T19:56:32.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.852+0000 7fa60633f700 1 -- 172.21.15.173:0/969829349 wait complete. 2024-01-21T19:56:32.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.852+0000 7fa60633f700 1 Processor -- start 2024-01-21T19:56:32.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.852+0000 7fa60633f700 1 -- start start 2024-01-21T19:56:32.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.852+0000 7fa60633f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6000f51a0 0x7fa6001064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:32.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.852+0000 7fa60633f700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6001069b0 con 0x7fa6000f51a0 2024-01-21T19:56:32.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.853+0000 7fa60533d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6000f51a0 0x7fa6001064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:32.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.853+0000 7fa60533d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6000f51a0 0x7fa6001064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:59182/0 (socket says 172.21.15.173:59182) 2024-01-21T19:56:32.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.853+0000 7fa60533d700 1 -- 172.21.15.173:0/411699241 learned_addr learned my addr 172.21.15.173:0/411699241 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:56:32.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.853+0000 7fa60533d700 1 -- 172.21.15.173:0/411699241 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5fc005040 con 0x7fa6000f51a0 2024-01-21T19:56:32.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.853+0000 7fa60533d700 1 --2- 172.21.15.173:0/411699241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6000f51a0 0x7fa6001064c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fa5fc013510 tx=0x7fa5fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:32.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.853+0000 7fa5f5ffb700 1 -- 172.21.15.173:0/411699241 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5fc009320 con 0x7fa6000f51a0 2024-01-21T19:56:32.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.853+0000 7fa5f5ffb700 1 -- 172.21.15.173:0/411699241 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa5fc013f00 con 0x7fa6000f51a0 2024-01-21T19:56:32.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.853+0000 7fa60633f700 1 -- 172.21.15.173:0/411699241 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa600106bb0 con 0x7fa6000f51a0 2024-01-21T19:56:32.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.853+0000 7fa5f5ffb700 1 -- 172.21.15.173:0/411699241 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5fc014dc0 con 0x7fa6000f51a0 2024-01-21T19:56:32.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.853+0000 7fa60633f700 1 -- 172.21.15.173:0/411699241 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa600107040 con 0x7fa6000f51a0 2024-01-21T19:56:32.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.854+0000 7fa5f5ffb700 1 -- 172.21.15.173:0/411699241 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa5fc0095f0 con 0x7fa6000f51a0 2024-01-21T19:56:32.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.854+0000 7fa60633f700 1 -- 172.21.15.173:0/411699241 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa600045ad0 con 0x7fa6000f51a0 2024-01-21T19:56:32.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.854+0000 7fa5f5ffb700 1 --2- 172.21.15.173:0/411699241 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa5ec02ff60 0x7fa5ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:32.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.854+0000 7fa5f5ffb700 1 -- 172.21.15.173:0/411699241 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa5fc0410c0 con 0x7fa6000f51a0 2024-01-21T19:56:32.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.855+0000 7fa604b3c700 1 --2- 172.21.15.173:0/411699241 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa5ec02ff60 0x7fa5ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:32.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.855+0000 7fa604b3c700 1 --2- 172.21.15.173:0/411699241 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa5ec02ff60 0x7fa5ec032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fa5f00096a0 tx=0x7fa5f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:32.851 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:32.859+0000 7fa5f5ffb700 1 -- 172.21.15.173:0/411699241 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa5fc014420 con 0x7fa6000f51a0 2024-01-21T19:56:33.056 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:33.064+0000 7fa60633f700 1 -- 172.21.15.173:0/411699241 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa600107e80 con 0x7fa6000f51a0 2024-01-21T19:56:33.057 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:33.065+0000 7fa5f5ffb700 1 -- 172.21.15.173:0/411699241 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa5fc010230 con 0x7fa6000f51a0 2024-01-21T19:56:33.057 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:56:33.057 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:56:33.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:33.066+0000 7fa60633f700 1 -- 172.21.15.173:0/411699241 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa5ec02ff60 msgr2=0x7fa5ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:33.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:33.066+0000 7fa60633f700 1 --2- 172.21.15.173:0/411699241 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa5ec02ff60 0x7fa5ec032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fa5f00096a0 tx=0x7fa5f0006b40).stop 2024-01-21T19:56:33.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:33.066+0000 7fa60633f700 1 -- 172.21.15.173:0/411699241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6000f51a0 msgr2=0x7fa6001064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:33.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:33.066+0000 7fa60633f700 1 --2- 172.21.15.173:0/411699241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6000f51a0 0x7fa6001064c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fa5fc013510 tx=0x7fa5fc00b040).stop 2024-01-21T19:56:33.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:33.066+0000 7fa60633f700 1 -- 172.21.15.173:0/411699241 shutdown_connections 2024-01-21T19:56:33.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:33.066+0000 7fa60633f700 1 --2- 172.21.15.173:0/411699241 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa5ec02ff60 0x7fa5ec032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:33.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:33.066+0000 7fa60633f700 1 --2- 172.21.15.173:0/411699241 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6000f51a0 0x7fa6001064c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:33.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:33.066+0000 7fa60633f700 1 -- 172.21.15.173:0/411699241 >> 172.21.15.173:0/411699241 conn(0x7fa6000f09a0 msgr2=0x7fa6000f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:33.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:33.066+0000 7fa60633f700 1 -- 172.21.15.173:0/411699241 shutdown_connections 2024-01-21T19:56:33.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:33.066+0000 7fa60633f700 1 -- 172.21.15.173:0/411699241 wait complete. 2024-01-21T19:56:33.060 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:33.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:32 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:31.010033+0000 mgr.smithi169.espgue (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:34.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:33 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:33.066514+0000 mon.smithi169 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.173:0/411699241' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:34.463 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:34.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:56:35.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:34 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:33.010277+0000 mgr.smithi169.espgue (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:35.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.629+0000 7f8bf50af700 1 -- 172.21.15.173:0/1255731693 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bf00f5410 msgr2=0x7f8bf00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:35.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.629+0000 7f8bf50af700 1 --2- 172.21.15.173:0/1255731693 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bf00f5410 0x7f8bf00f5830 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f8be4009fb0 tx=0x7f8be4009b30).stop 2024-01-21T19:56:35.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.629+0000 7f8bf50af700 1 -- 172.21.15.173:0/1255731693 shutdown_connections 2024-01-21T19:56:35.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.629+0000 7f8bf50af700 1 --2- 172.21.15.173:0/1255731693 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bf00f5410 0x7f8bf00f5830 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:35.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.629+0000 7f8bf50af700 1 -- 172.21.15.173:0/1255731693 >> 172.21.15.173:0/1255731693 conn(0x7f8bf00f0b70 msgr2=0x7f8bf00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:35.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.629+0000 7f8bf50af700 1 -- 172.21.15.173:0/1255731693 shutdown_connections 2024-01-21T19:56:35.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.629+0000 7f8bf50af700 1 -- 172.21.15.173:0/1255731693 wait complete. 2024-01-21T19:56:35.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.630+0000 7f8bf50af700 1 Processor -- start 2024-01-21T19:56:35.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.630+0000 7f8bf50af700 1 -- start start 2024-01-21T19:56:35.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.630+0000 7f8bf50af700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bf00f5410 0x7f8bf0106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:35.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.630+0000 7f8bf50af700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf0106c10 con 0x7f8bf00f5410 2024-01-21T19:56:35.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.630+0000 7f8beed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bf00f5410 0x7f8bf0106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:35.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.630+0000 7f8beed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bf00f5410 0x7f8bf0106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:45792/0 (socket says 172.21.15.173:45792) 2024-01-21T19:56:35.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.630+0000 7f8beed9d700 1 -- 172.21.15.173:0/2508520761 learned_addr learned my addr 172.21.15.173:0/2508520761 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:56:35.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.630+0000 7f8beed9d700 1 -- 172.21.15.173:0/2508520761 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be4005040 con 0x7f8bf00f5410 2024-01-21T19:56:35.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.630+0000 7f8beed9d700 1 --2- 172.21.15.173:0/2508520761 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bf00f5410 0x7f8bf0106720 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f8be4013510 tx=0x7f8be400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:35.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.631+0000 7f8bd77fe700 1 -- 172.21.15.173:0/2508520761 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8be4009320 con 0x7f8bf00f5410 2024-01-21T19:56:35.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.631+0000 7f8bf50af700 1 -- 172.21.15.173:0/2508520761 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf0106e10 con 0x7f8bf00f5410 2024-01-21T19:56:35.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.631+0000 7f8bd77fe700 1 -- 172.21.15.173:0/2508520761 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8be4013f00 con 0x7f8bf00f5410 2024-01-21T19:56:35.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.631+0000 7f8bd77fe700 1 -- 172.21.15.173:0/2508520761 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8be4014d90 con 0x7f8bf00f5410 2024-01-21T19:56:35.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.631+0000 7f8bf50af700 1 -- 172.21.15.173:0/2508520761 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf01072a0 con 0x7f8bf00f5410 2024-01-21T19:56:35.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.631+0000 7f8bd77fe700 1 -- 172.21.15.173:0/2508520761 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8be40095f0 con 0x7f8bf00f5410 2024-01-21T19:56:35.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.631+0000 7f8bf50af700 1 -- 172.21.15.173:0/2508520761 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf0045ad0 con 0x7f8bf00f5410 2024-01-21T19:56:35.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.632+0000 7f8bd77fe700 1 --2- 172.21.15.173:0/2508520761 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8bd802ffb0 0x7f8bd8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:35.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.632+0000 7f8bd77fe700 1 -- 172.21.15.173:0/2508520761 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8be4040f90 con 0x7f8bf00f5410 2024-01-21T19:56:35.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.632+0000 7f8bee59c700 1 --2- 172.21.15.173:0/2508520761 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8bd802ffb0 0x7f8bd8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:35.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.632+0000 7f8bee59c700 1 --2- 172.21.15.173:0/2508520761 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8bd802ffb0 0x7f8bd8032470 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f8be00096a0 tx=0x7f8be0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:35.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.636+0000 7f8bd77fe700 1 -- 172.21.15.173:0/2508520761 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8be401d7f0 con 0x7f8bf00f5410 2024-01-21T19:56:35.838 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.846+0000 7f8bf50af700 1 -- 172.21.15.173:0/2508520761 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8bf01080f0 con 0x7f8bf00f5410 2024-01-21T19:56:35.838 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.846+0000 7f8bd77fe700 1 -- 172.21.15.173:0/2508520761 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8be4010230 con 0x7f8bf00f5410 2024-01-21T19:56:35.839 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:56:35.839 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:56:35.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.848+0000 7f8bf50af700 1 -- 172.21.15.173:0/2508520761 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8bd802ffb0 msgr2=0x7f8bd8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:35.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.848+0000 7f8bf50af700 1 --2- 172.21.15.173:0/2508520761 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8bd802ffb0 0x7f8bd8032470 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f8be00096a0 tx=0x7f8be0006b40).stop 2024-01-21T19:56:35.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.848+0000 7f8bf50af700 1 -- 172.21.15.173:0/2508520761 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bf00f5410 msgr2=0x7f8bf0106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:35.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.848+0000 7f8bf50af700 1 --2- 172.21.15.173:0/2508520761 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bf00f5410 0x7f8bf0106720 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f8be4013510 tx=0x7f8be400b040).stop 2024-01-21T19:56:35.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.848+0000 7f8bf50af700 1 -- 172.21.15.173:0/2508520761 shutdown_connections 2024-01-21T19:56:35.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.848+0000 7f8bf50af700 1 --2- 172.21.15.173:0/2508520761 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f8bd802ffb0 0x7f8bd8032470 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:35.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.848+0000 7f8bf50af700 1 --2- 172.21.15.173:0/2508520761 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8bf00f5410 0x7f8bf0106720 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:35.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.848+0000 7f8bf50af700 1 -- 172.21.15.173:0/2508520761 >> 172.21.15.173:0/2508520761 conn(0x7f8bf00f0b70 msgr2=0x7f8bf005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:35.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.848+0000 7f8bf50af700 1 -- 172.21.15.173:0/2508520761 shutdown_connections 2024-01-21T19:56:35.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:35.848+0000 7f8bf50af700 1 -- 172.21.15.173:0/2508520761 wait complete. 2024-01-21T19:56:35.841 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:37.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:36 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:35.010572+0000 mgr.smithi169.espgue (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:37.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:36 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:35.848076+0000 mon.smithi169 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.173:0/2508520761' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:37.257 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:37.257 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:56:38.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.448+0000 7f20248d0700 1 -- 172.21.15.173:0/2105765664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20200f51a0 msgr2=0x7f20200f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:38.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.448+0000 7f20248d0700 1 --2- 172.21.15.173:0/2105765664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20200f51a0 0x7f20200f55c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f2010009fb0 tx=0x7f2010009b30).stop 2024-01-21T19:56:38.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.448+0000 7f20248d0700 1 -- 172.21.15.173:0/2105765664 shutdown_connections 2024-01-21T19:56:38.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.448+0000 7f20248d0700 1 --2- 172.21.15.173:0/2105765664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20200f51a0 0x7f20200f55c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:38.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.448+0000 7f20248d0700 1 -- 172.21.15.173:0/2105765664 >> 172.21.15.173:0/2105765664 conn(0x7f20200f09a0 msgr2=0x7f20200f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:38.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.449+0000 7f20248d0700 1 -- 172.21.15.173:0/2105765664 shutdown_connections 2024-01-21T19:56:38.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.449+0000 7f20248d0700 1 -- 172.21.15.173:0/2105765664 wait complete. 2024-01-21T19:56:38.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.449+0000 7f20248d0700 1 Processor -- start 2024-01-21T19:56:38.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.449+0000 7f20248d0700 1 -- start start 2024-01-21T19:56:38.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.449+0000 7f20248d0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20200f51a0 0x7f20201091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:38.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.449+0000 7f20248d0700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2020060240 con 0x7f20200f51a0 2024-01-21T19:56:38.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.450+0000 7f201ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20200f51a0 0x7f20201091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:38.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.450+0000 7f201ed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20200f51a0 0x7f20201091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:45810/0 (socket says 172.21.15.173:45810) 2024-01-21T19:56:38.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.450+0000 7f201ed9d700 1 -- 172.21.15.173:0/1031492562 learned_addr learned my addr 172.21.15.173:0/1031492562 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:56:38.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.450+0000 7f201ed9d700 1 -- 172.21.15.173:0/1031492562 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2010005040 con 0x7f20200f51a0 2024-01-21T19:56:38.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.450+0000 7f201ed9d700 1 --2- 172.21.15.173:0/1031492562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20200f51a0 0x7f20201091b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f2010013510 tx=0x7f201000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:38.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.450+0000 7f20077fe700 1 -- 172.21.15.173:0/1031492562 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2010009320 con 0x7f20200f51a0 2024-01-21T19:56:38.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.450+0000 7f20077fe700 1 -- 172.21.15.173:0/1031492562 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2010013f00 con 0x7f20200f51a0 2024-01-21T19:56:38.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.450+0000 7f20248d0700 1 -- 172.21.15.173:0/1031492562 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2020060440 con 0x7f20200f51a0 2024-01-21T19:56:38.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.450+0000 7f20077fe700 1 -- 172.21.15.173:0/1031492562 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2010014dc0 con 0x7f20200f51a0 2024-01-21T19:56:38.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.450+0000 7f20248d0700 1 -- 172.21.15.173:0/1031492562 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2020109860 con 0x7f20200f51a0 2024-01-21T19:56:38.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.451+0000 7f20077fe700 1 -- 172.21.15.173:0/1031492562 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f20100095f0 con 0x7f20200f51a0 2024-01-21T19:56:38.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.451+0000 7f20077fe700 1 --2- 172.21.15.173:0/1031492562 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f200802ff60 0x7f2008032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:38.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.451+0000 7f20248d0700 1 -- 172.21.15.173:0/1031492562 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2020045ad0 con 0x7f20200f51a0 2024-01-21T19:56:38.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.451+0000 7f20077fe700 1 -- 172.21.15.173:0/1031492562 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2010041040 con 0x7f20200f51a0 2024-01-21T19:56:38.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.451+0000 7f201e59c700 1 --2- 172.21.15.173:0/1031492562 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f200802ff60 0x7f2008032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:38.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.452+0000 7f201e59c700 1 --2- 172.21.15.173:0/1031492562 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f200802ff60 0x7f2008032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f20140096a0 tx=0x7f2014006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:38.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.456+0000 7f20077fe700 1 -- 172.21.15.173:0/1031492562 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2010014420 con 0x7f20200f51a0 2024-01-21T19:56:38.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.662+0000 7f20248d0700 1 -- 172.21.15.173:0/1031492562 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f202010a080 con 0x7f20200f51a0 2024-01-21T19:56:38.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.663+0000 7f20077fe700 1 -- 172.21.15.173:0/1031492562 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2010010230 con 0x7f20200f51a0 2024-01-21T19:56:38.655 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:56:38.655 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:56:38.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.664+0000 7f20248d0700 1 -- 172.21.15.173:0/1031492562 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f200802ff60 msgr2=0x7f2008032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:38.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.664+0000 7f20248d0700 1 --2- 172.21.15.173:0/1031492562 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f200802ff60 0x7f2008032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f20140096a0 tx=0x7f2014006b40).stop 2024-01-21T19:56:38.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.664+0000 7f20248d0700 1 -- 172.21.15.173:0/1031492562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20200f51a0 msgr2=0x7f20201091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:38.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.664+0000 7f20248d0700 1 --2- 172.21.15.173:0/1031492562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20200f51a0 0x7f20201091b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f2010013510 tx=0x7f201000b040).stop 2024-01-21T19:56:38.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.664+0000 7f20248d0700 1 -- 172.21.15.173:0/1031492562 shutdown_connections 2024-01-21T19:56:38.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.664+0000 7f20248d0700 1 --2- 172.21.15.173:0/1031492562 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f200802ff60 0x7f2008032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:38.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.664+0000 7f20248d0700 1 --2- 172.21.15.173:0/1031492562 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20200f51a0 0x7f20201091b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:38.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.664+0000 7f20248d0700 1 -- 172.21.15.173:0/1031492562 >> 172.21.15.173:0/1031492562 conn(0x7f20200f09a0 msgr2=0x7f20200f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:38.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.664+0000 7f20248d0700 1 -- 172.21.15.173:0/1031492562 shutdown_connections 2024-01-21T19:56:38.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:38.664+0000 7f20248d0700 1 -- 172.21.15.173:0/1031492562 wait complete. 2024-01-21T19:56:38.658 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:39.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:37.010912+0000 mgr.smithi169.espgue (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:39.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:38.664204+0000 mon.smithi169 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.173:0/1031492562' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:40.040 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:40.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:56:41.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:40 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:39.011252+0000 mgr.smithi169.espgue (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:41.286 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.292+0000 7fa6af37a700 1 -- 172.21.15.173:0/1051147706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6a80f51a0 msgr2=0x7fa6a80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:41.286 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.292+0000 7fa6af37a700 1 --2- 172.21.15.173:0/1051147706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6a80f51a0 0x7fa6a80f55c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fa698009fb0 tx=0x7fa698009b30).stop 2024-01-21T19:56:41.286 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.292+0000 7fa6af37a700 1 -- 172.21.15.173:0/1051147706 shutdown_connections 2024-01-21T19:56:41.286 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.292+0000 7fa6af37a700 1 --2- 172.21.15.173:0/1051147706 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6a80f51a0 0x7fa6a80f55c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:41.286 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.292+0000 7fa6af37a700 1 -- 172.21.15.173:0/1051147706 >> 172.21.15.173:0/1051147706 conn(0x7fa6a80f09a0 msgr2=0x7fa6a80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:41.286 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.292+0000 7fa6af37a700 1 -- 172.21.15.173:0/1051147706 shutdown_connections 2024-01-21T19:56:41.286 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.292+0000 7fa6af37a700 1 -- 172.21.15.173:0/1051147706 wait complete. 2024-01-21T19:56:41.286 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.292+0000 7fa6af37a700 1 Processor -- start 2024-01-21T19:56:41.286 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.293+0000 7fa6af37a700 1 -- start start 2024-01-21T19:56:41.286 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.293+0000 7fa6af37a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6a80f51a0 0x7fa6a81064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:41.287 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.293+0000 7fa6af37a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a81069d0 con 0x7fa6a80f51a0 2024-01-21T19:56:41.287 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.293+0000 7fa6ae378700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6a80f51a0 0x7fa6a81064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:41.287 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.293+0000 7fa6ae378700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6a80f51a0 0x7fa6a81064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:45834/0 (socket says 172.21.15.173:45834) 2024-01-21T19:56:41.287 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.293+0000 7fa6ae378700 1 -- 172.21.15.173:0/3195940041 learned_addr learned my addr 172.21.15.173:0/3195940041 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:56:41.287 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.293+0000 7fa6ae378700 1 -- 172.21.15.173:0/3195940041 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa698005040 con 0x7fa6a80f51a0 2024-01-21T19:56:41.287 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.293+0000 7fa6ae378700 1 --2- 172.21.15.173:0/3195940041 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6a80f51a0 0x7fa6a81064e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fa698013510 tx=0x7fa69800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:41.287 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.293+0000 7fa69effd700 1 -- 172.21.15.173:0/3195940041 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa698009320 con 0x7fa6a80f51a0 2024-01-21T19:56:41.287 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.293+0000 7fa6af37a700 1 -- 172.21.15.173:0/3195940041 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a8106bd0 con 0x7fa6a80f51a0 2024-01-21T19:56:41.287 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.293+0000 7fa69effd700 1 -- 172.21.15.173:0/3195940041 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa698013f00 con 0x7fa6a80f51a0 2024-01-21T19:56:41.287 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.293+0000 7fa69effd700 1 -- 172.21.15.173:0/3195940041 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa698014dc0 con 0x7fa6a80f51a0 2024-01-21T19:56:41.287 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.294+0000 7fa6af37a700 1 -- 172.21.15.173:0/3195940041 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a8107060 con 0x7fa6a80f51a0 2024-01-21T19:56:41.288 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.294+0000 7fa69effd700 1 -- 172.21.15.173:0/3195940041 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6980095f0 con 0x7fa6a80f51a0 2024-01-21T19:56:41.288 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.294+0000 7fa6af37a700 1 -- 172.21.15.173:0/3195940041 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a8045ad0 con 0x7fa6a80f51a0 2024-01-21T19:56:41.288 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.294+0000 7fa69effd700 1 --2- 172.21.15.173:0/3195940041 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa69402ff60 0x7fa694032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:41.288 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.295+0000 7fa69effd700 1 -- 172.21.15.173:0/3195940041 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa6980410c0 con 0x7fa6a80f51a0 2024-01-21T19:56:41.288 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.295+0000 7fa6adb77700 1 --2- 172.21.15.173:0/3195940041 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa69402ff60 0x7fa694032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:41.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.295+0000 7fa6adb77700 1 --2- 172.21.15.173:0/3195940041 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa69402ff60 0x7fa694032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fa6a40096a0 tx=0x7fa6a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:41.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.299+0000 7fa69effd700 1 -- 172.21.15.173:0/3195940041 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa69801c790 con 0x7fa6a80f51a0 2024-01-21T19:56:41.495 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.503+0000 7fa6af37a700 1 -- 172.21.15.173:0/3195940041 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6a8107f10 con 0x7fa6a80f51a0 2024-01-21T19:56:41.496 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.504+0000 7fa69effd700 1 -- 172.21.15.173:0/3195940041 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa698010230 con 0x7fa6a80f51a0 2024-01-21T19:56:41.496 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:56:41.496 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:56:41.498 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.505+0000 7fa6af37a700 1 -- 172.21.15.173:0/3195940041 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa69402ff60 msgr2=0x7fa694032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:41.498 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.505+0000 7fa6af37a700 1 --2- 172.21.15.173:0/3195940041 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa69402ff60 0x7fa694032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fa6a40096a0 tx=0x7fa6a4006b40).stop 2024-01-21T19:56:41.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.505+0000 7fa6af37a700 1 -- 172.21.15.173:0/3195940041 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6a80f51a0 msgr2=0x7fa6a81064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:41.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.505+0000 7fa6af37a700 1 --2- 172.21.15.173:0/3195940041 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6a80f51a0 0x7fa6a81064e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fa698013510 tx=0x7fa69800b040).stop 2024-01-21T19:56:41.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.505+0000 7fa6af37a700 1 -- 172.21.15.173:0/3195940041 shutdown_connections 2024-01-21T19:56:41.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.505+0000 7fa6af37a700 1 --2- 172.21.15.173:0/3195940041 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa69402ff60 0x7fa694032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:41.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.505+0000 7fa6af37a700 1 --2- 172.21.15.173:0/3195940041 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa6a80f51a0 0x7fa6a81064e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:41.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.505+0000 7fa6af37a700 1 -- 172.21.15.173:0/3195940041 >> 172.21.15.173:0/3195940041 conn(0x7fa6a80f09a0 msgr2=0x7fa6a805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:41.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.505+0000 7fa6af37a700 1 -- 172.21.15.173:0/3195940041 shutdown_connections 2024-01-21T19:56:41.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:41.505+0000 7fa6af37a700 1 -- 172.21.15.173:0/3195940041 wait complete. 2024-01-21T19:56:41.499 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:42.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:41 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:41.505251+0000 mon.smithi169 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.173:0/3195940041' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:42.899 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:42.899 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:56:43.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:42 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:41.011474+0000 mgr.smithi169.espgue (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:44.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.151+0000 7f3397ede700 1 -- 172.21.15.173:0/332582656 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33900f6510 msgr2=0x7f33900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:44.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.151+0000 7f3397ede700 1 --2- 172.21.15.173:0/332582656 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33900f6510 0x7f33900f6930 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f3384009fb0 tx=0x7f3384009b30).stop 2024-01-21T19:56:44.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.151+0000 7f3397ede700 1 -- 172.21.15.173:0/332582656 shutdown_connections 2024-01-21T19:56:44.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.151+0000 7f3397ede700 1 --2- 172.21.15.173:0/332582656 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33900f6510 0x7f33900f6930 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:44.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.151+0000 7f3397ede700 1 -- 172.21.15.173:0/332582656 >> 172.21.15.173:0/332582656 conn(0x7f33900f1c70 msgr2=0x7f33900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:44.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.151+0000 7f3397ede700 1 -- 172.21.15.173:0/332582656 shutdown_connections 2024-01-21T19:56:44.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.151+0000 7f3397ede700 1 -- 172.21.15.173:0/332582656 wait complete. 2024-01-21T19:56:44.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.152+0000 7f3397ede700 1 Processor -- start 2024-01-21T19:56:44.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.152+0000 7f3397ede700 1 -- start start 2024-01-21T19:56:44.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.152+0000 7f3397ede700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33900f6510 0x7f339006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:44.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.152+0000 7f3397ede700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3390069e20 con 0x7f33900f6510 2024-01-21T19:56:44.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.152+0000 7f3395c7a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33900f6510 0x7f339006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:44.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.152+0000 7f3395c7a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33900f6510 0x7f339006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:36294/0 (socket says 172.21.15.173:36294) 2024-01-21T19:56:44.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.152+0000 7f3395c7a700 1 -- 172.21.15.173:0/2406235496 learned_addr learned my addr 172.21.15.173:0/2406235496 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:56:44.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.153+0000 7f3395c7a700 1 -- 172.21.15.173:0/2406235496 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3384005040 con 0x7f33900f6510 2024-01-21T19:56:44.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.153+0000 7f3395c7a700 1 --2- 172.21.15.173:0/2406235496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33900f6510 0x7f339006b9a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f3384013510 tx=0x7f338400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:44.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.153+0000 7f33827fc700 1 -- 172.21.15.173:0/2406235496 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3384009320 con 0x7f33900f6510 2024-01-21T19:56:44.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.153+0000 7f33827fc700 1 -- 172.21.15.173:0/2406235496 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3384013f00 con 0x7f33900f6510 2024-01-21T19:56:44.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.153+0000 7f3397ede700 1 -- 172.21.15.173:0/2406235496 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3390069fc0 con 0x7f33900f6510 2024-01-21T19:56:44.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.153+0000 7f33827fc700 1 -- 172.21.15.173:0/2406235496 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3384014d90 con 0x7f33900f6510 2024-01-21T19:56:44.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.153+0000 7f3397ede700 1 -- 172.21.15.173:0/2406235496 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f339006a450 con 0x7f33900f6510 2024-01-21T19:56:44.148 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.154+0000 7f33827fc700 1 -- 172.21.15.173:0/2406235496 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f33840095f0 con 0x7f33900f6510 2024-01-21T19:56:44.148 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.154+0000 7f3397ede700 1 -- 172.21.15.173:0/2406235496 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3390059100 con 0x7f33900f6510 2024-01-21T19:56:44.148 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.154+0000 7f33827fc700 1 --2- 172.21.15.173:0/2406235496 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f337c02ff60 0x7f337c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:44.148 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.154+0000 7f33827fc700 1 -- 172.21.15.173:0/2406235496 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f33840410c0 con 0x7f33900f6510 2024-01-21T19:56:44.149 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.154+0000 7f3395479700 1 --2- 172.21.15.173:0/2406235496 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f337c02ff60 0x7f337c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:44.149 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.155+0000 7f3395479700 1 --2- 172.21.15.173:0/2406235496 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f337c02ff60 0x7f337c032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f338c0096a0 tx=0x7f338c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:44.151 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.159+0000 7f33827fc700 1 -- 172.21.15.173:0/2406235496 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3384014420 con 0x7f33900f6510 2024-01-21T19:56:44.355 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.363+0000 7f3397ede700 1 -- 172.21.15.173:0/2406235496 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f33900fbf60 con 0x7f33900f6510 2024-01-21T19:56:44.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.364+0000 7f33827fc700 1 -- 172.21.15.173:0/2406235496 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f338401d7f0 con 0x7f33900f6510 2024-01-21T19:56:44.356 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:56:44.356 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:56:44.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.366+0000 7f3397ede700 1 -- 172.21.15.173:0/2406235496 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f337c02ff60 msgr2=0x7f337c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:44.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.366+0000 7f3397ede700 1 --2- 172.21.15.173:0/2406235496 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f337c02ff60 0x7f337c032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f338c0096a0 tx=0x7f338c006b40).stop 2024-01-21T19:56:44.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.366+0000 7f3397ede700 1 -- 172.21.15.173:0/2406235496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33900f6510 msgr2=0x7f339006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:44.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.366+0000 7f3397ede700 1 --2- 172.21.15.173:0/2406235496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33900f6510 0x7f339006b9a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f3384013510 tx=0x7f338400b040).stop 2024-01-21T19:56:44.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.366+0000 7f3397ede700 1 -- 172.21.15.173:0/2406235496 shutdown_connections 2024-01-21T19:56:44.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.366+0000 7f3397ede700 1 --2- 172.21.15.173:0/2406235496 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f337c02ff60 0x7f337c032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:44.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.366+0000 7f3397ede700 1 --2- 172.21.15.173:0/2406235496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f33900f6510 0x7f339006b9a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:44.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.366+0000 7f3397ede700 1 -- 172.21.15.173:0/2406235496 >> 172.21.15.173:0/2406235496 conn(0x7f33900f1c70 msgr2=0x7f33900f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:44.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.366+0000 7f3397ede700 1 -- 172.21.15.173:0/2406235496 shutdown_connections 2024-01-21T19:56:44.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:44.366+0000 7f3397ede700 1 -- 172.21.15.173:0/2406235496 wait complete. 2024-01-21T19:56:44.359 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:45.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:44 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:43.011720+0000 mgr.smithi169.espgue (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:45.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:44 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:44.365702+0000 mon.smithi169 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.173:0/2406235496' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:45.750 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:45.750 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:56:46.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.907+0000 7f62f9104700 1 -- 172.21.15.173:0/2937690599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62f40f6510 msgr2=0x7f62f40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:46.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.907+0000 7f62f9104700 1 --2- 172.21.15.173:0/2937690599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62f40f6510 0x7f62f40f6930 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f62e4009fb0 tx=0x7f62e4009b30).stop 2024-01-21T19:56:46.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.907+0000 7f62f9104700 1 -- 172.21.15.173:0/2937690599 shutdown_connections 2024-01-21T19:56:46.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.907+0000 7f62f9104700 1 --2- 172.21.15.173:0/2937690599 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62f40f6510 0x7f62f40f6930 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:46.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.907+0000 7f62f9104700 1 -- 172.21.15.173:0/2937690599 >> 172.21.15.173:0/2937690599 conn(0x7f62f40f1c70 msgr2=0x7f62f40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:46.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.907+0000 7f62f9104700 1 -- 172.21.15.173:0/2937690599 shutdown_connections 2024-01-21T19:56:46.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.907+0000 7f62f9104700 1 -- 172.21.15.173:0/2937690599 wait complete. 2024-01-21T19:56:46.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.908+0000 7f62f9104700 1 Processor -- start 2024-01-21T19:56:46.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.908+0000 7f62f9104700 1 -- start start 2024-01-21T19:56:46.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.908+0000 7f62f9104700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62f40f6510 0x7f62f4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:46.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.908+0000 7f62f9104700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f4106c20 con 0x7f62f40f6510 2024-01-21T19:56:46.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.909+0000 7f62f2d9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62f40f6510 0x7f62f4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:46.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.909+0000 7f62f2d9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62f40f6510 0x7f62f4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:36316/0 (socket says 172.21.15.173:36316) 2024-01-21T19:56:46.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.909+0000 7f62f2d9d700 1 -- 172.21.15.173:0/547729909 learned_addr learned my addr 172.21.15.173:0/547729909 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:56:46.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.909+0000 7f62f2d9d700 1 -- 172.21.15.173:0/547729909 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e4005040 con 0x7f62f40f6510 2024-01-21T19:56:46.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.909+0000 7f62f2d9d700 1 --2- 172.21.15.173:0/547729909 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62f40f6510 0x7f62f4106730 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f62e4013510 tx=0x7f62e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:46.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.909+0000 7f62eb7fe700 1 -- 172.21.15.173:0/547729909 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62e4009320 con 0x7f62f40f6510 2024-01-21T19:56:46.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.909+0000 7f62f9104700 1 -- 172.21.15.173:0/547729909 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f4106e20 con 0x7f62f40f6510 2024-01-21T19:56:46.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.909+0000 7f62eb7fe700 1 -- 172.21.15.173:0/547729909 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62e4013f00 con 0x7f62f40f6510 2024-01-21T19:56:46.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.909+0000 7f62eb7fe700 1 -- 172.21.15.173:0/547729909 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62e4014d90 con 0x7f62f40f6510 2024-01-21T19:56:46.902 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.909+0000 7f62f9104700 1 -- 172.21.15.173:0/547729909 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f41072b0 con 0x7f62f40f6510 2024-01-21T19:56:46.902 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.910+0000 7f62eb7fe700 1 -- 172.21.15.173:0/547729909 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f62e40095f0 con 0x7f62f40f6510 2024-01-21T19:56:46.903 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.910+0000 7f62f9104700 1 -- 172.21.15.173:0/547729909 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f4045ad0 con 0x7f62f40f6510 2024-01-21T19:56:46.903 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.910+0000 7f62eb7fe700 1 --2- 172.21.15.173:0/547729909 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f62e002ff60 0x7f62e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:46.903 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.910+0000 7f62eb7fe700 1 -- 172.21.15.173:0/547729909 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f62e40410c0 con 0x7f62f40f6510 2024-01-21T19:56:46.903 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.910+0000 7f62f259c700 1 --2- 172.21.15.173:0/547729909 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f62e002ff60 0x7f62e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:46.903 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.911+0000 7f62f259c700 1 --2- 172.21.15.173:0/547729909 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f62e002ff60 0x7f62e0032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f62dc0096a0 tx=0x7f62dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:46.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:46.915+0000 7f62eb7fe700 1 -- 172.21.15.173:0/547729909 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62e4014420 con 0x7f62f40f6510 2024-01-21T19:56:47.113 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:47.122+0000 7f62f9104700 1 -- 172.21.15.173:0/547729909 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f62f40fbf60 con 0x7f62f40f6510 2024-01-21T19:56:47.114 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:47.123+0000 7f62eb7fe700 1 -- 172.21.15.173:0/547729909 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f62e4010230 con 0x7f62f40f6510 2024-01-21T19:56:47.115 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:56:47.115 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:56:47.116 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:47.124+0000 7f62f9104700 1 -- 172.21.15.173:0/547729909 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f62e002ff60 msgr2=0x7f62e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:47.116 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:47.124+0000 7f62f9104700 1 --2- 172.21.15.173:0/547729909 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f62e002ff60 0x7f62e0032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f62dc0096a0 tx=0x7f62dc006b40).stop 2024-01-21T19:56:47.117 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:47.124+0000 7f62f9104700 1 -- 172.21.15.173:0/547729909 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62f40f6510 msgr2=0x7f62f4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:47.117 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:47.124+0000 7f62f9104700 1 --2- 172.21.15.173:0/547729909 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62f40f6510 0x7f62f4106730 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f62e4013510 tx=0x7f62e400b040).stop 2024-01-21T19:56:47.117 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:47.124+0000 7f62f9104700 1 -- 172.21.15.173:0/547729909 shutdown_connections 2024-01-21T19:56:47.117 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:47.124+0000 7f62f9104700 1 --2- 172.21.15.173:0/547729909 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f62e002ff60 0x7f62e0032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:47.117 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:47.124+0000 7f62f9104700 1 --2- 172.21.15.173:0/547729909 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62f40f6510 0x7f62f4106730 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:47.117 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:47.124+0000 7f62f9104700 1 -- 172.21.15.173:0/547729909 >> 172.21.15.173:0/547729909 conn(0x7f62f40f1c70 msgr2=0x7f62f40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:47.117 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:47.124+0000 7f62f9104700 1 -- 172.21.15.173:0/547729909 shutdown_connections 2024-01-21T19:56:47.118 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:47.124+0000 7f62f9104700 1 -- 172.21.15.173:0/547729909 wait complete. 2024-01-21T19:56:47.118 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:47.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:46 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:45.012040+0000 mgr.smithi169.espgue (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:48.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:47 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:47.123576+0000 mon.smithi169 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.173:0/547729909' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:48.514 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:48.514 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:56:49.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:48 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:47.012326+0000 mgr.smithi169.espgue (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:49.717 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.725+0000 7fe8e6580700 1 -- 172.21.15.173:0/1332836342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8e0060280 msgr2=0x7fe8e00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:49.717 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.725+0000 7fe8e6580700 1 --2- 172.21.15.173:0/1332836342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8e0060280 0x7fe8e00f90f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fe8d0009fb0 tx=0x7fe8d0009b30).stop 2024-01-21T19:56:49.717 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.725+0000 7fe8e6580700 1 -- 172.21.15.173:0/1332836342 shutdown_connections 2024-01-21T19:56:49.717 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.725+0000 7fe8e6580700 1 --2- 172.21.15.173:0/1332836342 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8e0060280 0x7fe8e00f90f0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:49.717 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.725+0000 7fe8e6580700 1 -- 172.21.15.173:0/1332836342 >> 172.21.15.173:0/1332836342 conn(0x7fe8e00f0b70 msgr2=0x7fe8e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:49.717 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.725+0000 7fe8e6580700 1 -- 172.21.15.173:0/1332836342 shutdown_connections 2024-01-21T19:56:49.717 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.725+0000 7fe8e6580700 1 -- 172.21.15.173:0/1332836342 wait complete. 2024-01-21T19:56:49.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.726+0000 7fe8e6580700 1 Processor -- start 2024-01-21T19:56:49.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.726+0000 7fe8e6580700 1 -- start start 2024-01-21T19:56:49.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.726+0000 7fe8e6580700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8e0060280 0x7fe8e01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:49.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.726+0000 7fe8e6580700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e0106bd0 con 0x7fe8e0060280 2024-01-21T19:56:49.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.727+0000 7fe8dffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8e0060280 0x7fe8e01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:49.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.727+0000 7fe8dffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8e0060280 0x7fe8e01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:36326/0 (socket says 172.21.15.173:36326) 2024-01-21T19:56:49.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.727+0000 7fe8dffff700 1 -- 172.21.15.173:0/2258476324 learned_addr learned my addr 172.21.15.173:0/2258476324 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:56:49.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.727+0000 7fe8dffff700 1 -- 172.21.15.173:0/2258476324 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d0005040 con 0x7fe8e0060280 2024-01-21T19:56:49.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.727+0000 7fe8dffff700 1 --2- 172.21.15.173:0/2258476324 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8e0060280 0x7fe8e01066e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fe8d0013510 tx=0x7fe8d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:49.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.727+0000 7fe8dcff9700 1 -- 172.21.15.173:0/2258476324 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8d0009320 con 0x7fe8e0060280 2024-01-21T19:56:49.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.727+0000 7fe8e6580700 1 -- 172.21.15.173:0/2258476324 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e0106dd0 con 0x7fe8e0060280 2024-01-21T19:56:49.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.727+0000 7fe8dcff9700 1 -- 172.21.15.173:0/2258476324 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe8d0013f00 con 0x7fe8e0060280 2024-01-21T19:56:49.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.727+0000 7fe8dcff9700 1 -- 172.21.15.173:0/2258476324 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8d0014dc0 con 0x7fe8e0060280 2024-01-21T19:56:49.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.727+0000 7fe8e6580700 1 -- 172.21.15.173:0/2258476324 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e0107260 con 0x7fe8e0060280 2024-01-21T19:56:49.720 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.728+0000 7fe8dcff9700 1 -- 172.21.15.173:0/2258476324 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8d001d3d0 con 0x7fe8e0060280 2024-01-21T19:56:49.720 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.728+0000 7fe8dcff9700 1 --2- 172.21.15.173:0/2258476324 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe8c802ff60 0x7fe8c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:49.720 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.728+0000 7fe8e6580700 1 -- 172.21.15.173:0/2258476324 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e0045ad0 con 0x7fe8e0060280 2024-01-21T19:56:49.721 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.728+0000 7fe8dcff9700 1 -- 172.21.15.173:0/2258476324 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe8d0041320 con 0x7fe8e0060280 2024-01-21T19:56:49.721 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.728+0000 7fe8df7fe700 1 --2- 172.21.15.173:0/2258476324 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe8c802ff60 0x7fe8c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:49.721 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.729+0000 7fe8df7fe700 1 --2- 172.21.15.173:0/2258476324 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe8c802ff60 0x7fe8c8032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fe8d40096a0 tx=0x7fe8d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:49.724 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.733+0000 7fe8dcff9700 1 -- 172.21.15.173:0/2258476324 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe8d0012920 con 0x7fe8e0060280 2024-01-21T19:56:49.928 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.937+0000 7fe8e6580700 1 -- 172.21.15.173:0/2258476324 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8e0108110 con 0x7fe8e0060280 2024-01-21T19:56:49.929 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.938+0000 7fe8dcff9700 1 -- 172.21.15.173:0/2258476324 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe8d003a030 con 0x7fe8e0060280 2024-01-21T19:56:49.929 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:56:49.929 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:56:49.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.939+0000 7fe8e6580700 1 -- 172.21.15.173:0/2258476324 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe8c802ff60 msgr2=0x7fe8c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:49.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.939+0000 7fe8e6580700 1 --2- 172.21.15.173:0/2258476324 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe8c802ff60 0x7fe8c8032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fe8d40096a0 tx=0x7fe8d4006b40).stop 2024-01-21T19:56:49.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.939+0000 7fe8e6580700 1 -- 172.21.15.173:0/2258476324 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8e0060280 msgr2=0x7fe8e01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:49.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.939+0000 7fe8e6580700 1 --2- 172.21.15.173:0/2258476324 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8e0060280 0x7fe8e01066e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fe8d0013510 tx=0x7fe8d000b040).stop 2024-01-21T19:56:49.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.939+0000 7fe8e6580700 1 -- 172.21.15.173:0/2258476324 shutdown_connections 2024-01-21T19:56:49.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.939+0000 7fe8e6580700 1 --2- 172.21.15.173:0/2258476324 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe8c802ff60 0x7fe8c8032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:49.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.939+0000 7fe8e6580700 1 --2- 172.21.15.173:0/2258476324 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe8e0060280 0x7fe8e01066e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:49.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.939+0000 7fe8e6580700 1 -- 172.21.15.173:0/2258476324 >> 172.21.15.173:0/2258476324 conn(0x7fe8e00f0b70 msgr2=0x7fe8e00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:49.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.939+0000 7fe8e6580700 1 -- 172.21.15.173:0/2258476324 shutdown_connections 2024-01-21T19:56:49.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:49.939+0000 7fe8e6580700 1 -- 172.21.15.173:0/2258476324 wait complete. 2024-01-21T19:56:49.932 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:51.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:50 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:49.012663+0000 mgr.smithi169.espgue (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:51.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:50 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:49.938493+0000 mon.smithi169 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.173:0/2258476324' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:51.314 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:51.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:56:52.509 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.516+0000 7f3f92615700 1 -- 172.21.15.173:0/3254795750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f8c0f5410 msgr2=0x7f3f8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:52.509 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.516+0000 7f3f92615700 1 --2- 172.21.15.173:0/3254795750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f8c0f5410 0x7f3f8c0f5830 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f3f7c009fb0 tx=0x7f3f7c009b30).stop 2024-01-21T19:56:52.509 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.516+0000 7f3f92615700 1 -- 172.21.15.173:0/3254795750 shutdown_connections 2024-01-21T19:56:52.509 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.516+0000 7f3f92615700 1 --2- 172.21.15.173:0/3254795750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f8c0f5410 0x7f3f8c0f5830 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:52.509 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.516+0000 7f3f92615700 1 -- 172.21.15.173:0/3254795750 >> 172.21.15.173:0/3254795750 conn(0x7f3f8c0f0b70 msgr2=0x7f3f8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:52.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.516+0000 7f3f92615700 1 -- 172.21.15.173:0/3254795750 shutdown_connections 2024-01-21T19:56:52.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.516+0000 7f3f92615700 1 -- 172.21.15.173:0/3254795750 wait complete. 2024-01-21T19:56:52.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.517+0000 7f3f92615700 1 Processor -- start 2024-01-21T19:56:52.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.517+0000 7f3f92615700 1 -- start start 2024-01-21T19:56:52.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.517+0000 7f3f92615700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f8c0f5410 0x7f3f8c1055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:52.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.517+0000 7f3f92615700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c105ac0 con 0x7f3f8c0f5410 2024-01-21T19:56:52.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.517+0000 7f3f8bfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f8c0f5410 0x7f3f8c1055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:52.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.517+0000 7f3f8bfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f8c0f5410 0x7f3f8c1055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:36338/0 (socket says 172.21.15.173:36338) 2024-01-21T19:56:52.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.517+0000 7f3f8bfff700 1 -- 172.21.15.173:0/195332657 learned_addr learned my addr 172.21.15.173:0/195332657 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:56:52.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.518+0000 7f3f8bfff700 1 -- 172.21.15.173:0/195332657 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c005040 con 0x7f3f8c0f5410 2024-01-21T19:56:52.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.518+0000 7f3f8bfff700 1 --2- 172.21.15.173:0/195332657 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f8c0f5410 0x7f3f8c1055d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f3f7c013510 tx=0x7f3f7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:52.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.518+0000 7f3f88ff9700 1 -- 172.21.15.173:0/195332657 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f7c009320 con 0x7f3f8c0f5410 2024-01-21T19:56:52.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.518+0000 7f3f92615700 1 -- 172.21.15.173:0/195332657 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f8c105cc0 con 0x7f3f8c0f5410 2024-01-21T19:56:52.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.518+0000 7f3f88ff9700 1 -- 172.21.15.173:0/195332657 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f7c013f00 con 0x7f3f8c0f5410 2024-01-21T19:56:52.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.518+0000 7f3f88ff9700 1 -- 172.21.15.173:0/195332657 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f7c014d90 con 0x7f3f8c0f5410 2024-01-21T19:56:52.511 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.518+0000 7f3f92615700 1 -- 172.21.15.173:0/195332657 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f8c106150 con 0x7f3f8c0f5410 2024-01-21T19:56:52.511 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.519+0000 7f3f88ff9700 1 -- 172.21.15.173:0/195332657 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3f7c00f410 con 0x7f3f8c0f5410 2024-01-21T19:56:52.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.519+0000 7f3f88ff9700 1 --2- 172.21.15.173:0/195332657 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3f7802ff60 0x7f3f78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:52.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.519+0000 7f3f88ff9700 1 -- 172.21.15.173:0/195332657 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3f7c042d90 con 0x7f3f8c0f5410 2024-01-21T19:56:52.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.519+0000 7f3f92615700 1 -- 172.21.15.173:0/195332657 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f8c045ad0 con 0x7f3f8c0f5410 2024-01-21T19:56:52.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.519+0000 7f3f8b7fe700 1 --2- 172.21.15.173:0/195332657 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3f7802ff60 0x7f3f78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:52.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.519+0000 7f3f8b7fe700 1 --2- 172.21.15.173:0/195332657 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3f7802ff60 0x7f3f78032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f3f740096a0 tx=0x7f3f74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:52.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.523+0000 7f3f88ff9700 1 -- 172.21.15.173:0/195332657 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f7c0283a0 con 0x7f3f8c0f5410 2024-01-21T19:56:52.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.728+0000 7f3f92615700 1 -- 172.21.15.173:0/195332657 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f8c106ea0 con 0x7f3f8c0f5410 2024-01-21T19:56:52.720 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.729+0000 7f3f88ff9700 1 -- 172.21.15.173:0/195332657 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3f7c011290 con 0x7f3f8c0f5410 2024-01-21T19:56:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:56:52.720 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:56:52.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.731+0000 7f3f92615700 1 -- 172.21.15.173:0/195332657 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3f7802ff60 msgr2=0x7f3f78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:52.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.731+0000 7f3f92615700 1 --2- 172.21.15.173:0/195332657 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3f7802ff60 0x7f3f78032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f3f740096a0 tx=0x7f3f74006b40).stop 2024-01-21T19:56:52.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.731+0000 7f3f92615700 1 -- 172.21.15.173:0/195332657 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f8c0f5410 msgr2=0x7f3f8c1055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:52.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.731+0000 7f3f92615700 1 --2- 172.21.15.173:0/195332657 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f8c0f5410 0x7f3f8c1055d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f3f7c013510 tx=0x7f3f7c00b040).stop 2024-01-21T19:56:52.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.731+0000 7f3f92615700 1 -- 172.21.15.173:0/195332657 shutdown_connections 2024-01-21T19:56:52.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.731+0000 7f3f92615700 1 --2- 172.21.15.173:0/195332657 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3f7802ff60 0x7f3f78032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:52.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.731+0000 7f3f92615700 1 --2- 172.21.15.173:0/195332657 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3f8c0f5410 0x7f3f8c1055d0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:52.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.731+0000 7f3f92615700 1 -- 172.21.15.173:0/195332657 >> 172.21.15.173:0/195332657 conn(0x7f3f8c0f0b70 msgr2=0x7f3f8c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:52.723 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.731+0000 7f3f92615700 1 -- 172.21.15.173:0/195332657 shutdown_connections 2024-01-21T19:56:52.723 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:52.731+0000 7f3f92615700 1 -- 172.21.15.173:0/195332657 wait complete. 2024-01-21T19:56:52.723 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:53.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:52 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:51.013078+0000 mgr.smithi169.espgue (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:53.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:52 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:52.729824+0000 mon.smithi169 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.173:0/195332657' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:54.107 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:54.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:56:55.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:54 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:53.013427+0000 mgr.smithi169.espgue (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:55.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.222+0000 7f9fcc9e0700 1 -- 172.21.15.173:0/565627656 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fc80f5410 msgr2=0x7f9fc80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:55.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.222+0000 7f9fcc9e0700 1 --2- 172.21.15.173:0/565627656 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fc80f5410 0x7f9fc80f5830 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f9fb0009fb0 tx=0x7f9fb0009b30).stop 2024-01-21T19:56:55.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.222+0000 7f9fcc9e0700 1 -- 172.21.15.173:0/565627656 shutdown_connections 2024-01-21T19:56:55.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.222+0000 7f9fcc9e0700 1 --2- 172.21.15.173:0/565627656 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fc80f5410 0x7f9fc80f5830 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:55.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.222+0000 7f9fcc9e0700 1 -- 172.21.15.173:0/565627656 >> 172.21.15.173:0/565627656 conn(0x7f9fc80f0b70 msgr2=0x7f9fc80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:55.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.222+0000 7f9fcc9e0700 1 -- 172.21.15.173:0/565627656 shutdown_connections 2024-01-21T19:56:55.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.222+0000 7f9fcc9e0700 1 -- 172.21.15.173:0/565627656 wait complete. 2024-01-21T19:56:55.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.223+0000 7f9fcc9e0700 1 Processor -- start 2024-01-21T19:56:55.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.223+0000 7f9fcc9e0700 1 -- start start 2024-01-21T19:56:55.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.223+0000 7f9fcc9e0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fc80f5410 0x7f9fc81093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:55.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.223+0000 7f9fcc9e0700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc8060240 con 0x7f9fc80f5410 2024-01-21T19:56:55.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.223+0000 7f9fc659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fc80f5410 0x7f9fc81093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:55.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.223+0000 7f9fc659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fc80f5410 0x7f9fc81093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:60610/0 (socket says 172.21.15.173:60610) 2024-01-21T19:56:55.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.223+0000 7f9fc659c700 1 -- 172.21.15.173:0/1061796142 learned_addr learned my addr 172.21.15.173:0/1061796142 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:56:55.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.224+0000 7f9fc659c700 1 -- 172.21.15.173:0/1061796142 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb0005040 con 0x7f9fc80f5410 2024-01-21T19:56:55.216 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.224+0000 7f9fc659c700 1 --2- 172.21.15.173:0/1061796142 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fc80f5410 0x7f9fc81093d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f9fb0013510 tx=0x7f9fb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:55.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.224+0000 7f9fbeffd700 1 -- 172.21.15.173:0/1061796142 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fb0009320 con 0x7f9fc80f5410 2024-01-21T19:56:55.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.224+0000 7f9fcc9e0700 1 -- 172.21.15.173:0/1061796142 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fc8060440 con 0x7f9fc80f5410 2024-01-21T19:56:55.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.224+0000 7f9fbeffd700 1 -- 172.21.15.173:0/1061796142 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9fb0013f00 con 0x7f9fc80f5410 2024-01-21T19:56:55.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.224+0000 7f9fbeffd700 1 -- 172.21.15.173:0/1061796142 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fb0014d90 con 0x7f9fc80f5410 2024-01-21T19:56:55.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.224+0000 7f9fcc9e0700 1 -- 172.21.15.173:0/1061796142 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fc8109a80 con 0x7f9fc80f5410 2024-01-21T19:56:55.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.225+0000 7f9fbeffd700 1 -- 172.21.15.173:0/1061796142 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9fb00263a0 con 0x7f9fc80f5410 2024-01-21T19:56:55.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.225+0000 7f9fbeffd700 1 --2- 172.21.15.173:0/1061796142 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9fb402ff60 0x7f9fb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:55.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.225+0000 7f9fbeffd700 1 -- 172.21.15.173:0/1061796142 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9fb00410c0 con 0x7f9fc80f5410 2024-01-21T19:56:55.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.225+0000 7f9fcc9e0700 1 -- 172.21.15.173:0/1061796142 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fc8045ad0 con 0x7f9fc80f5410 2024-01-21T19:56:55.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.225+0000 7f9fc5d9b700 1 --2- 172.21.15.173:0/1061796142 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9fb402ff60 0x7f9fb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:55.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.226+0000 7f9fc5d9b700 1 --2- 172.21.15.173:0/1061796142 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9fb402ff60 0x7f9fb4032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f9fb80096a0 tx=0x7f9fb8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:55.221 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.230+0000 7f9fbeffd700 1 -- 172.21.15.173:0/1061796142 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9fb0012920 con 0x7f9fc80f5410 2024-01-21T19:56:55.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.439+0000 7f9fcc9e0700 1 -- 172.21.15.173:0/1061796142 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9fc80fae60 con 0x7f9fc80f5410 2024-01-21T19:56:55.431 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.440+0000 7f9fbeffd700 1 -- 172.21.15.173:0/1061796142 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9fb002b020 con 0x7f9fc80f5410 2024-01-21T19:56:55.432 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:56:55.432 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:56:55.433 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.442+0000 7f9fcc9e0700 1 -- 172.21.15.173:0/1061796142 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9fb402ff60 msgr2=0x7f9fb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:55.433 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.442+0000 7f9fcc9e0700 1 --2- 172.21.15.173:0/1061796142 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9fb402ff60 0x7f9fb4032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f9fb80096a0 tx=0x7f9fb8006b40).stop 2024-01-21T19:56:55.433 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.442+0000 7f9fcc9e0700 1 -- 172.21.15.173:0/1061796142 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fc80f5410 msgr2=0x7f9fc81093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:55.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.442+0000 7f9fcc9e0700 1 --2- 172.21.15.173:0/1061796142 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fc80f5410 0x7f9fc81093d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f9fb0013510 tx=0x7f9fb000b040).stop 2024-01-21T19:56:55.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.442+0000 7f9fcc9e0700 1 -- 172.21.15.173:0/1061796142 shutdown_connections 2024-01-21T19:56:55.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.442+0000 7f9fcc9e0700 1 --2- 172.21.15.173:0/1061796142 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9fb402ff60 0x7f9fb4032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:55.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.442+0000 7f9fcc9e0700 1 --2- 172.21.15.173:0/1061796142 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fc80f5410 0x7f9fc81093d0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:55.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.442+0000 7f9fcc9e0700 1 -- 172.21.15.173:0/1061796142 >> 172.21.15.173:0/1061796142 conn(0x7f9fc80f0b70 msgr2=0x7f9fc80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:55.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.442+0000 7f9fcc9e0700 1 -- 172.21.15.173:0/1061796142 shutdown_connections 2024-01-21T19:56:55.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:55.442+0000 7f9fcc9e0700 1 -- 172.21.15.173:0/1061796142 wait complete. 2024-01-21T19:56:55.434 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:56.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:55 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:55.440962+0000 mon.smithi169 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.173:0/1061796142' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:56.826 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:56.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:56:57.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:56 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:55.013721+0000 mgr.smithi169.espgue (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:58.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.014+0000 7f3af010d700 1 -- 172.21.15.173:0/289059530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ae80f5410 msgr2=0x7f3ae80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:58.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.014+0000 7f3af010d700 1 --2- 172.21.15.173:0/289059530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ae80f5410 0x7f3ae80f5830 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f3ad8009fb0 tx=0x7f3ad8009b30).stop 2024-01-21T19:56:58.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.014+0000 7f3af010d700 1 -- 172.21.15.173:0/289059530 shutdown_connections 2024-01-21T19:56:58.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.014+0000 7f3af010d700 1 --2- 172.21.15.173:0/289059530 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ae80f5410 0x7f3ae80f5830 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:58.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.014+0000 7f3af010d700 1 -- 172.21.15.173:0/289059530 >> 172.21.15.173:0/289059530 conn(0x7f3ae80f0b70 msgr2=0x7f3ae80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:58.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.014+0000 7f3af010d700 1 -- 172.21.15.173:0/289059530 shutdown_connections 2024-01-21T19:56:58.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.014+0000 7f3af010d700 1 -- 172.21.15.173:0/289059530 wait complete. 2024-01-21T19:56:58.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.015+0000 7f3af010d700 1 Processor -- start 2024-01-21T19:56:58.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.015+0000 7f3af010d700 1 -- start start 2024-01-21T19:56:58.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.015+0000 7f3af010d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ae80f5410 0x7f3ae8106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:58.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.015+0000 7f3af010d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae8106c10 con 0x7f3ae80f5410 2024-01-21T19:56:58.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.015+0000 7f3aedea9700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ae80f5410 0x7f3ae8106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:58.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.015+0000 7f3aedea9700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ae80f5410 0x7f3ae8106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:60626/0 (socket says 172.21.15.173:60626) 2024-01-21T19:56:58.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.015+0000 7f3aedea9700 1 -- 172.21.15.173:0/1692183256 learned_addr learned my addr 172.21.15.173:0/1692183256 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:56:58.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.015+0000 7f3aedea9700 1 -- 172.21.15.173:0/1692183256 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad8005040 con 0x7f3ae80f5410 2024-01-21T19:56:58.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.016+0000 7f3aedea9700 1 --2- 172.21.15.173:0/1692183256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ae80f5410 0x7f3ae8106720 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f3ad8014000 tx=0x7f3ad800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:58.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.016+0000 7f3ade7fc700 1 -- 172.21.15.173:0/1692183256 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ad80095f0 con 0x7f3ae80f5410 2024-01-21T19:56:58.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.016+0000 7f3af010d700 1 -- 172.21.15.173:0/1692183256 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae8106e10 con 0x7f3ae80f5410 2024-01-21T19:56:58.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.016+0000 7f3ade7fc700 1 -- 172.21.15.173:0/1692183256 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ad8007e70 con 0x7f3ae80f5410 2024-01-21T19:56:58.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.016+0000 7f3ade7fc700 1 -- 172.21.15.173:0/1692183256 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ad8014ea0 con 0x7f3ae80f5410 2024-01-21T19:56:58.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.016+0000 7f3af010d700 1 -- 172.21.15.173:0/1692183256 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae81072a0 con 0x7f3ae80f5410 2024-01-21T19:56:58.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.017+0000 7f3ade7fc700 1 -- 172.21.15.173:0/1692183256 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ad8025070 con 0x7f3ae80f5410 2024-01-21T19:56:58.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.017+0000 7f3ade7fc700 1 --2- 172.21.15.173:0/1692183256 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3ad402ff60 0x7f3ad4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:56:58.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.017+0000 7f3af010d700 1 -- 172.21.15.173:0/1692183256 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae8045ad0 con 0x7f3ae80f5410 2024-01-21T19:56:58.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.017+0000 7f3ade7fc700 1 -- 172.21.15.173:0/1692183256 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ad8040980 con 0x7f3ae80f5410 2024-01-21T19:56:58.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.017+0000 7f3aed6a8700 1 --2- 172.21.15.173:0/1692183256 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3ad402ff60 0x7f3ad4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:56:58.010 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.017+0000 7f3aed6a8700 1 --2- 172.21.15.173:0/1692183256 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3ad402ff60 0x7f3ad4032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f3ae40096a0 tx=0x7f3ae4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:56:58.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.021+0000 7f3ade7fc700 1 -- 172.21.15.173:0/1692183256 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ad80079c0 con 0x7f3ae80f5410 2024-01-21T19:56:58.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.228+0000 7f3af010d700 1 -- 172.21.15.173:0/1692183256 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ae8108150 con 0x7f3ae80f5410 2024-01-21T19:56:58.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.228+0000 7f3ade7fc700 1 -- 172.21.15.173:0/1692183256 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3ad800f300 con 0x7f3ae80f5410 2024-01-21T19:56:58.220 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:56:58.220 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:56:58.221 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.230+0000 7f3af010d700 1 -- 172.21.15.173:0/1692183256 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3ad402ff60 msgr2=0x7f3ad4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:58.221 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.230+0000 7f3af010d700 1 --2- 172.21.15.173:0/1692183256 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3ad402ff60 0x7f3ad4032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f3ae40096a0 tx=0x7f3ae4006b40).stop 2024-01-21T19:56:58.221 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.230+0000 7f3af010d700 1 -- 172.21.15.173:0/1692183256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ae80f5410 msgr2=0x7f3ae8106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:56:58.221 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.230+0000 7f3af010d700 1 --2- 172.21.15.173:0/1692183256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ae80f5410 0x7f3ae8106720 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f3ad8014000 tx=0x7f3ad800b040).stop 2024-01-21T19:56:58.221 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.230+0000 7f3af010d700 1 -- 172.21.15.173:0/1692183256 shutdown_connections 2024-01-21T19:56:58.221 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.230+0000 7f3af010d700 1 --2- 172.21.15.173:0/1692183256 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f3ad402ff60 0x7f3ad4032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:58.221 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.230+0000 7f3af010d700 1 --2- 172.21.15.173:0/1692183256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3ae80f5410 0x7f3ae8106720 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:56:58.221 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.230+0000 7f3af010d700 1 -- 172.21.15.173:0/1692183256 >> 172.21.15.173:0/1692183256 conn(0x7f3ae80f0b70 msgr2=0x7f3ae805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:56:58.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.230+0000 7f3af010d700 1 -- 172.21.15.173:0/1692183256 shutdown_connections 2024-01-21T19:56:58.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:56:58.230+0000 7f3af010d700 1 -- 172.21.15.173:0/1692183256 wait complete. 2024-01-21T19:56:58.222 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:56:59.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:57.014093+0000 mgr.smithi169.espgue (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:56:59.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:56:58 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:56:58.229090+0000 mon.smithi169 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.173:0/1692183256' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:56:59.625 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:56:59.625 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:57:00.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.893+0000 7f12e717d700 1 -- 172.21.15.173:0/3992115573 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e00f62a0 msgr2=0x7f12e00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:00.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.893+0000 7f12e717d700 1 --2- 172.21.15.173:0/3992115573 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e00f62a0 0x7f12e00f66c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f12dc009fb0 tx=0x7f12dc009b30).stop 2024-01-21T19:57:00.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.893+0000 7f12e717d700 1 -- 172.21.15.173:0/3992115573 shutdown_connections 2024-01-21T19:57:00.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.893+0000 7f12e717d700 1 --2- 172.21.15.173:0/3992115573 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e00f62a0 0x7f12e00f66c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:00.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.893+0000 7f12e717d700 1 -- 172.21.15.173:0/3992115573 >> 172.21.15.173:0/3992115573 conn(0x7f12e00f1ac0 msgr2=0x7f12e00f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:00.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.893+0000 7f12e717d700 1 -- 172.21.15.173:0/3992115573 shutdown_connections 2024-01-21T19:57:00.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.893+0000 7f12e717d700 1 -- 172.21.15.173:0/3992115573 wait complete. 2024-01-21T19:57:00.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.893+0000 7f12e717d700 1 Processor -- start 2024-01-21T19:57:00.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.893+0000 7f12e717d700 1 -- start start 2024-01-21T19:57:00.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.894+0000 7f12e717d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e00f62a0 0x7f12e0106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:00.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.894+0000 7f12e717d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e01069f0 con 0x7f12e00f62a0 2024-01-21T19:57:00.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.894+0000 7f12e617b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e00f62a0 0x7f12e0106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:00.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.894+0000 7f12e617b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e00f62a0 0x7f12e0106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:60634/0 (socket says 172.21.15.173:60634) 2024-01-21T19:57:00.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.894+0000 7f12e617b700 1 -- 172.21.15.173:0/3670502307 learned_addr learned my addr 172.21.15.173:0/3670502307 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:57:00.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.894+0000 7f12e617b700 1 -- 172.21.15.173:0/3670502307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12dc005040 con 0x7f12e00f62a0 2024-01-21T19:57:00.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.894+0000 7f12e617b700 1 --2- 172.21.15.173:0/3670502307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e00f62a0 0x7f12e0106500 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f12dc013510 tx=0x7f12dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:00.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.894+0000 7f12d6ffd700 1 -- 172.21.15.173:0/3670502307 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12dc009320 con 0x7f12e00f62a0 2024-01-21T19:57:00.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.894+0000 7f12e717d700 1 -- 172.21.15.173:0/3670502307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e0106bf0 con 0x7f12e00f62a0 2024-01-21T19:57:00.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.894+0000 7f12d6ffd700 1 -- 172.21.15.173:0/3670502307 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12dc013f00 con 0x7f12e00f62a0 2024-01-21T19:57:00.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.894+0000 7f12d6ffd700 1 -- 172.21.15.173:0/3670502307 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12dc014dc0 con 0x7f12e00f62a0 2024-01-21T19:57:00.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.895+0000 7f12e717d700 1 -- 172.21.15.173:0/3670502307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e0107080 con 0x7f12e00f62a0 2024-01-21T19:57:00.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.895+0000 7f12d6ffd700 1 -- 172.21.15.173:0/3670502307 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12dc0095f0 con 0x7f12e00f62a0 2024-01-21T19:57:00.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.895+0000 7f12e717d700 1 -- 172.21.15.173:0/3670502307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e0045ad0 con 0x7f12e00f62a0 2024-01-21T19:57:00.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.895+0000 7f12d6ffd700 1 --2- 172.21.15.173:0/3670502307 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f12cc02ff60 0x7f12cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:00.888 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.895+0000 7f12d6ffd700 1 -- 172.21.15.173:0/3670502307 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12dc041000 con 0x7f12e00f62a0 2024-01-21T19:57:00.888 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.896+0000 7f12e597a700 1 --2- 172.21.15.173:0/3670502307 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f12cc02ff60 0x7f12cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:00.888 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.896+0000 7f12e597a700 1 --2- 172.21.15.173:0/3670502307 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f12cc02ff60 0x7f12cc032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f12d00096a0 tx=0x7f12d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:00.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:00.900+0000 7f12d6ffd700 1 -- 172.21.15.173:0/3670502307 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12dc014420 con 0x7f12e00f62a0 2024-01-21T19:57:01.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:01.103+0000 7f12e717d700 1 -- 172.21.15.173:0/3670502307 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12e0107f30 con 0x7f12e00f62a0 2024-01-21T19:57:01.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:01.104+0000 7f12d6ffd700 1 -- 172.21.15.173:0/3670502307 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f12dc010230 con 0x7f12e00f62a0 2024-01-21T19:57:01.095 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:57:01.095 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:57:01.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:01.105+0000 7f12e717d700 1 -- 172.21.15.173:0/3670502307 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f12cc02ff60 msgr2=0x7f12cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:01.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:01.105+0000 7f12e717d700 1 --2- 172.21.15.173:0/3670502307 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f12cc02ff60 0x7f12cc032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f12d00096a0 tx=0x7f12d0006b40).stop 2024-01-21T19:57:01.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:01.105+0000 7f12e717d700 1 -- 172.21.15.173:0/3670502307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e00f62a0 msgr2=0x7f12e0106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:01.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:01.105+0000 7f12e717d700 1 --2- 172.21.15.173:0/3670502307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e00f62a0 0x7f12e0106500 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f12dc013510 tx=0x7f12dc00b040).stop 2024-01-21T19:57:01.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:01.105+0000 7f12e717d700 1 -- 172.21.15.173:0/3670502307 shutdown_connections 2024-01-21T19:57:01.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:01.105+0000 7f12e717d700 1 --2- 172.21.15.173:0/3670502307 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f12cc02ff60 0x7f12cc032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:01.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:01.105+0000 7f12e717d700 1 --2- 172.21.15.173:0/3670502307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e00f62a0 0x7f12e0106500 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:01.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:01.105+0000 7f12e717d700 1 -- 172.21.15.173:0/3670502307 >> 172.21.15.173:0/3670502307 conn(0x7f12e00f1ac0 msgr2=0x7f12e00f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:01.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:01.105+0000 7f12e717d700 1 -- 172.21.15.173:0/3670502307 shutdown_connections 2024-01-21T19:57:01.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:01.105+0000 7f12e717d700 1 -- 172.21.15.173:0/3670502307 wait complete. 2024-01-21T19:57:01.098 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:57:01.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:00 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:56:59.014455+0000 mgr.smithi169.espgue (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:57:02.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:01 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:57:01.104566+0000 mon.smithi169 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.173:0/3670502307' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:57:02.580 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:57:02.580 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:57:03.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:02 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:57:01.014733+0000 mgr.smithi169.espgue (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:57:03.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.778+0000 7f6c8d10f700 1 -- 172.21.15.173:0/1059764065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c880f51a0 msgr2=0x7f6c880f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:03.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.778+0000 7f6c8d10f700 1 --2- 172.21.15.173:0/1059764065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c880f51a0 0x7f6c880f55c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f6c78009fb0 tx=0x7f6c78009b30).stop 2024-01-21T19:57:03.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.778+0000 7f6c8d10f700 1 -- 172.21.15.173:0/1059764065 shutdown_connections 2024-01-21T19:57:03.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.778+0000 7f6c8d10f700 1 --2- 172.21.15.173:0/1059764065 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c880f51a0 0x7f6c880f55c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:03.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.778+0000 7f6c8d10f700 1 -- 172.21.15.173:0/1059764065 >> 172.21.15.173:0/1059764065 conn(0x7f6c880f09a0 msgr2=0x7f6c880f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:03.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.778+0000 7f6c8d10f700 1 -- 172.21.15.173:0/1059764065 shutdown_connections 2024-01-21T19:57:03.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.778+0000 7f6c8d10f700 1 -- 172.21.15.173:0/1059764065 wait complete. 2024-01-21T19:57:03.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.778+0000 7f6c8d10f700 1 Processor -- start 2024-01-21T19:57:03.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.779+0000 7f6c8d10f700 1 -- start start 2024-01-21T19:57:03.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.779+0000 7f6c8d10f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c880f51a0 0x7f6c881053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:03.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.779+0000 7f6c8d10f700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c881058a0 con 0x7f6c880f51a0 2024-01-21T19:57:03.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.779+0000 7f6c87fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c880f51a0 0x7f6c881053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:03.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.779+0000 7f6c87fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c880f51a0 0x7f6c881053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:48534/0 (socket says 172.21.15.173:48534) 2024-01-21T19:57:03.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.779+0000 7f6c87fff700 1 -- 172.21.15.173:0/2210700884 learned_addr learned my addr 172.21.15.173:0/2210700884 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:57:03.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.779+0000 7f6c87fff700 1 -- 172.21.15.173:0/2210700884 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c78005040 con 0x7f6c880f51a0 2024-01-21T19:57:03.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.779+0000 7f6c87fff700 1 --2- 172.21.15.173:0/2210700884 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c880f51a0 0x7f6c881053b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f6c78013510 tx=0x7f6c7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:03.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.780+0000 7f6c84ff9700 1 -- 172.21.15.173:0/2210700884 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c78009320 con 0x7f6c880f51a0 2024-01-21T19:57:03.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.780+0000 7f6c8d10f700 1 -- 172.21.15.173:0/2210700884 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c88105aa0 con 0x7f6c880f51a0 2024-01-21T19:57:03.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.780+0000 7f6c84ff9700 1 -- 172.21.15.173:0/2210700884 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c78013f00 con 0x7f6c880f51a0 2024-01-21T19:57:03.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.780+0000 7f6c84ff9700 1 -- 172.21.15.173:0/2210700884 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c78014dc0 con 0x7f6c880f51a0 2024-01-21T19:57:03.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.780+0000 7f6c8d10f700 1 -- 172.21.15.173:0/2210700884 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c88105f30 con 0x7f6c880f51a0 2024-01-21T19:57:03.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.781+0000 7f6c84ff9700 1 -- 172.21.15.173:0/2210700884 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c780095f0 con 0x7f6c880f51a0 2024-01-21T19:57:03.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.781+0000 7f6c8d10f700 1 -- 172.21.15.173:0/2210700884 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c88045ad0 con 0x7f6c880f51a0 2024-01-21T19:57:03.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.781+0000 7f6c84ff9700 1 --2- 172.21.15.173:0/2210700884 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f6c7002ff60 0x7f6c70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:03.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.781+0000 7f6c84ff9700 1 -- 172.21.15.173:0/2210700884 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c78041020 con 0x7f6c880f51a0 2024-01-21T19:57:03.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.781+0000 7f6c877fe700 1 --2- 172.21.15.173:0/2210700884 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f6c7002ff60 0x7f6c70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:03.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.781+0000 7f6c877fe700 1 --2- 172.21.15.173:0/2210700884 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f6c7002ff60 0x7f6c70032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f6c7c0096a0 tx=0x7f6c7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:03.776 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.786+0000 7f6c84ff9700 1 -- 172.21.15.173:0/2210700884 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c78014420 con 0x7f6c880f51a0 2024-01-21T19:57:03.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.991+0000 7f6c8d10f700 1 -- 172.21.15.173:0/2210700884 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c88106d80 con 0x7f6c880f51a0 2024-01-21T19:57:03.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.992+0000 7f6c84ff9700 1 -- 172.21.15.173:0/2210700884 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6c7801c920 con 0x7f6c880f51a0 2024-01-21T19:57:03.984 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:57:03.984 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:57:03.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.994+0000 7f6c8d10f700 1 -- 172.21.15.173:0/2210700884 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f6c7002ff60 msgr2=0x7f6c70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:03.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.994+0000 7f6c8d10f700 1 --2- 172.21.15.173:0/2210700884 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f6c7002ff60 0x7f6c70032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f6c7c0096a0 tx=0x7f6c7c006b40).stop 2024-01-21T19:57:03.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.994+0000 7f6c8d10f700 1 -- 172.21.15.173:0/2210700884 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c880f51a0 msgr2=0x7f6c881053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:03.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.994+0000 7f6c8d10f700 1 --2- 172.21.15.173:0/2210700884 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c880f51a0 0x7f6c881053b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f6c78013510 tx=0x7f6c7800b040).stop 2024-01-21T19:57:03.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.994+0000 7f6c8d10f700 1 -- 172.21.15.173:0/2210700884 shutdown_connections 2024-01-21T19:57:03.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.994+0000 7f6c8d10f700 1 --2- 172.21.15.173:0/2210700884 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f6c7002ff60 0x7f6c70032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:03.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.994+0000 7f6c8d10f700 1 --2- 172.21.15.173:0/2210700884 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c880f51a0 0x7f6c881053b0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:03.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.994+0000 7f6c8d10f700 1 -- 172.21.15.173:0/2210700884 >> 172.21.15.173:0/2210700884 conn(0x7f6c880f09a0 msgr2=0x7f6c880f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:03.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.994+0000 7f6c8d10f700 1 -- 172.21.15.173:0/2210700884 shutdown_connections 2024-01-21T19:57:03.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:03.994+0000 7f6c8d10f700 1 -- 172.21.15.173:0/2210700884 wait complete. 2024-01-21T19:57:03.986 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:57:05.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:04 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:57:03.015090+0000 mgr.smithi169.espgue (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:57:05.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:04 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:57:03.992926+0000 mon.smithi169 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.173:0/2210700884' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:57:05.367 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:57:05.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:57:06.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.590+0000 7fceaa4f4700 1 -- 172.21.15.173:0/3818315443 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcea4060220 msgr2=0x7fcea40f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:06.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.590+0000 7fceaa4f4700 1 --2- 172.21.15.173:0/3818315443 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcea4060220 0x7fcea40f9150 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fce94009fb0 tx=0x7fce94009b30).stop 2024-01-21T19:57:06.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.590+0000 7fceaa4f4700 1 -- 172.21.15.173:0/3818315443 shutdown_connections 2024-01-21T19:57:06.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.590+0000 7fceaa4f4700 1 --2- 172.21.15.173:0/3818315443 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcea4060220 0x7fcea40f9150 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:06.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.590+0000 7fceaa4f4700 1 -- 172.21.15.173:0/3818315443 >> 172.21.15.173:0/3818315443 conn(0x7fcea40f0b70 msgr2=0x7fcea40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:06.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.590+0000 7fceaa4f4700 1 -- 172.21.15.173:0/3818315443 shutdown_connections 2024-01-21T19:57:06.583 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.590+0000 7fceaa4f4700 1 -- 172.21.15.173:0/3818315443 wait complete. 2024-01-21T19:57:06.583 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.591+0000 7fceaa4f4700 1 Processor -- start 2024-01-21T19:57:06.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.591+0000 7fceaa4f4700 1 -- start start 2024-01-21T19:57:06.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.591+0000 7fceaa4f4700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcea4060220 0x7fcea41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:06.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.591+0000 7fceaa4f4700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea4106bd0 con 0x7fcea4060220 2024-01-21T19:57:06.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.591+0000 7fcea3fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcea4060220 0x7fcea41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:06.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.591+0000 7fcea3fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcea4060220 0x7fcea41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:48538/0 (socket says 172.21.15.173:48538) 2024-01-21T19:57:06.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.591+0000 7fcea3fff700 1 -- 172.21.15.173:0/2456344487 learned_addr learned my addr 172.21.15.173:0/2456344487 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:57:06.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.592+0000 7fcea3fff700 1 -- 172.21.15.173:0/2456344487 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce94005040 con 0x7fcea4060220 2024-01-21T19:57:06.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.592+0000 7fcea3fff700 1 --2- 172.21.15.173:0/2456344487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcea4060220 0x7fcea41066e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fce94013510 tx=0x7fce9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:06.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.592+0000 7fcea0ff9700 1 -- 172.21.15.173:0/2456344487 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce94009320 con 0x7fcea4060220 2024-01-21T19:57:06.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.592+0000 7fceaa4f4700 1 -- 172.21.15.173:0/2456344487 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea4106dd0 con 0x7fcea4060220 2024-01-21T19:57:06.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.592+0000 7fcea0ff9700 1 -- 172.21.15.173:0/2456344487 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce94013f00 con 0x7fcea4060220 2024-01-21T19:57:06.586 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.592+0000 7fcea0ff9700 1 -- 172.21.15.173:0/2456344487 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce94014dc0 con 0x7fcea4060220 2024-01-21T19:57:06.586 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.592+0000 7fceaa4f4700 1 -- 172.21.15.173:0/2456344487 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea4107260 con 0x7fcea4060220 2024-01-21T19:57:06.586 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.593+0000 7fcea0ff9700 1 -- 172.21.15.173:0/2456344487 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce94014420 con 0x7fcea4060220 2024-01-21T19:57:06.586 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.593+0000 7fceaa4f4700 1 -- 172.21.15.173:0/2456344487 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea4045ad0 con 0x7fcea4060220 2024-01-21T19:57:06.586 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.593+0000 7fcea0ff9700 1 --2- 172.21.15.173:0/2456344487 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fce8c02ff60 0x7fce8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:06.586 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.593+0000 7fcea0ff9700 1 -- 172.21.15.173:0/2456344487 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce940410d0 con 0x7fcea4060220 2024-01-21T19:57:06.586 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.593+0000 7fcea37fe700 1 --2- 172.21.15.173:0/2456344487 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fce8c02ff60 0x7fce8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:06.586 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.594+0000 7fcea37fe700 1 --2- 172.21.15.173:0/2456344487 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fce8c02ff60 0x7fce8c032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fce980096a0 tx=0x7fce98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:06.589 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.598+0000 7fcea0ff9700 1 -- 172.21.15.173:0/2456344487 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce940095f0 con 0x7fcea4060220 2024-01-21T19:57:06.794 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.803+0000 7fceaa4f4700 1 -- 172.21.15.173:0/2456344487 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcea4108110 con 0x7fcea4060220 2024-01-21T19:57:06.795 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.804+0000 7fcea0ff9700 1 -- 172.21.15.173:0/2456344487 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fce9401cb10 con 0x7fcea4060220 2024-01-21T19:57:06.795 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:57:06.795 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:57:06.796 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.805+0000 7fceaa4f4700 1 -- 172.21.15.173:0/2456344487 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fce8c02ff60 msgr2=0x7fce8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:06.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.805+0000 7fceaa4f4700 1 --2- 172.21.15.173:0/2456344487 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fce8c02ff60 0x7fce8c032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fce980096a0 tx=0x7fce98006b40).stop 2024-01-21T19:57:06.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.805+0000 7fceaa4f4700 1 -- 172.21.15.173:0/2456344487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcea4060220 msgr2=0x7fcea41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:06.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.805+0000 7fceaa4f4700 1 --2- 172.21.15.173:0/2456344487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcea4060220 0x7fcea41066e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fce94013510 tx=0x7fce9400b040).stop 2024-01-21T19:57:06.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.805+0000 7fceaa4f4700 1 -- 172.21.15.173:0/2456344487 shutdown_connections 2024-01-21T19:57:06.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.805+0000 7fceaa4f4700 1 --2- 172.21.15.173:0/2456344487 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fce8c02ff60 0x7fce8c032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:06.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.805+0000 7fceaa4f4700 1 --2- 172.21.15.173:0/2456344487 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcea4060220 0x7fcea41066e0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:06.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.805+0000 7fceaa4f4700 1 -- 172.21.15.173:0/2456344487 >> 172.21.15.173:0/2456344487 conn(0x7fcea40f0b70 msgr2=0x7fcea40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:06.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.805+0000 7fceaa4f4700 1 -- 172.21.15.173:0/2456344487 shutdown_connections 2024-01-21T19:57:06.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:06.805+0000 7fceaa4f4700 1 -- 172.21.15.173:0/2456344487 wait complete. 2024-01-21T19:57:06.797 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:57:07.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:06 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:57:05.015431+0000 mgr.smithi169.espgue (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:57:08.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:07 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:57:06.804528+0000 mon.smithi169 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.173:0/2456344487' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:57:08.181 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:57:08.181 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:57:09.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:08 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:57:07.015793+0000 mgr.smithi169.espgue (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:57:09.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.384+0000 7f2d2b7d2700 1 -- 172.21.15.173:0/151769291 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d240f5410 msgr2=0x7f2d240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:09.376 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.384+0000 7f2d2b7d2700 1 --2- 172.21.15.173:0/151769291 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d240f5410 0x7f2d240f5830 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f2d18009fb0 tx=0x7f2d18009b30).stop 2024-01-21T19:57:09.376 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.384+0000 7f2d2b7d2700 1 -- 172.21.15.173:0/151769291 shutdown_connections 2024-01-21T19:57:09.376 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.384+0000 7f2d2b7d2700 1 --2- 172.21.15.173:0/151769291 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d240f5410 0x7f2d240f5830 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:09.376 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.384+0000 7f2d2b7d2700 1 -- 172.21.15.173:0/151769291 >> 172.21.15.173:0/151769291 conn(0x7f2d240f0b70 msgr2=0x7f2d240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:09.376 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.384+0000 7f2d2b7d2700 1 -- 172.21.15.173:0/151769291 shutdown_connections 2024-01-21T19:57:09.376 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.384+0000 7f2d2b7d2700 1 -- 172.21.15.173:0/151769291 wait complete. 2024-01-21T19:57:09.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.385+0000 7f2d2b7d2700 1 Processor -- start 2024-01-21T19:57:09.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.385+0000 7f2d2b7d2700 1 -- start start 2024-01-21T19:57:09.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.385+0000 7f2d2b7d2700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d240f5410 0x7f2d24105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:09.377 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.385+0000 7f2d2b7d2700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d24105b20 con 0x7f2d240f5410 2024-01-21T19:57:09.378 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.385+0000 7f2d2956e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d240f5410 0x7f2d24105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:09.378 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.385+0000 7f2d2956e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d240f5410 0x7f2d24105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:48548/0 (socket says 172.21.15.173:48548) 2024-01-21T19:57:09.378 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.385+0000 7f2d2956e700 1 -- 172.21.15.173:0/2582343870 learned_addr learned my addr 172.21.15.173:0/2582343870 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:57:09.378 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.386+0000 7f2d2956e700 1 -- 172.21.15.173:0/2582343870 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d18005040 con 0x7f2d240f5410 2024-01-21T19:57:09.378 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.386+0000 7f2d2956e700 1 --2- 172.21.15.173:0/2582343870 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d240f5410 0x7f2d24105630 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f2d18013510 tx=0x7f2d1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:09.378 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.386+0000 7f2d15ffb700 1 -- 172.21.15.173:0/2582343870 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d18009320 con 0x7f2d240f5410 2024-01-21T19:57:09.378 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.386+0000 7f2d2b7d2700 1 -- 172.21.15.173:0/2582343870 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d24105d20 con 0x7f2d240f5410 2024-01-21T19:57:09.378 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.386+0000 7f2d15ffb700 1 -- 172.21.15.173:0/2582343870 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d18013f00 con 0x7f2d240f5410 2024-01-21T19:57:09.378 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.386+0000 7f2d15ffb700 1 -- 172.21.15.173:0/2582343870 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d18014d90 con 0x7f2d240f5410 2024-01-21T19:57:09.378 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.386+0000 7f2d2b7d2700 1 -- 172.21.15.173:0/2582343870 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d241061b0 con 0x7f2d240f5410 2024-01-21T19:57:09.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.387+0000 7f2d15ffb700 1 -- 172.21.15.173:0/2582343870 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d180095f0 con 0x7f2d240f5410 2024-01-21T19:57:09.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.387+0000 7f2d15ffb700 1 --2- 172.21.15.173:0/2582343870 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2d1002ff60 0x7f2d10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:09.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.387+0000 7f2d2b7d2700 1 -- 172.21.15.173:0/2582343870 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d24045ad0 con 0x7f2d240f5410 2024-01-21T19:57:09.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.387+0000 7f2d15ffb700 1 -- 172.21.15.173:0/2582343870 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d18040fe0 con 0x7f2d240f5410 2024-01-21T19:57:09.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.387+0000 7f2d28d6d700 1 --2- 172.21.15.173:0/2582343870 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2d1002ff60 0x7f2d10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:09.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.388+0000 7f2d28d6d700 1 --2- 172.21.15.173:0/2582343870 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2d1002ff60 0x7f2d10032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f2d200096a0 tx=0x7f2d20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:09.383 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.392+0000 7f2d15ffb700 1 -- 172.21.15.173:0/2582343870 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d1806d2b0 con 0x7f2d240f5410 2024-01-21T19:57:09.588 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.596+0000 7f2d2b7d2700 1 -- 172.21.15.173:0/2582343870 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d240faf50 con 0x7f2d240f5410 2024-01-21T19:57:09.588 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.597+0000 7f2d15ffb700 1 -- 172.21.15.173:0/2582343870 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2d1801d7f0 con 0x7f2d240f5410 2024-01-21T19:57:09.589 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:57:09.589 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:57:09.590 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.598+0000 7f2d2b7d2700 1 -- 172.21.15.173:0/2582343870 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2d1002ff60 msgr2=0x7f2d10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:09.590 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.598+0000 7f2d2b7d2700 1 --2- 172.21.15.173:0/2582343870 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2d1002ff60 0x7f2d10032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f2d200096a0 tx=0x7f2d20006b40).stop 2024-01-21T19:57:09.590 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.598+0000 7f2d2b7d2700 1 -- 172.21.15.173:0/2582343870 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d240f5410 msgr2=0x7f2d24105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:09.590 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.598+0000 7f2d2b7d2700 1 --2- 172.21.15.173:0/2582343870 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d240f5410 0x7f2d24105630 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f2d18013510 tx=0x7f2d1800b040).stop 2024-01-21T19:57:09.590 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.598+0000 7f2d2b7d2700 1 -- 172.21.15.173:0/2582343870 shutdown_connections 2024-01-21T19:57:09.590 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.598+0000 7f2d2b7d2700 1 --2- 172.21.15.173:0/2582343870 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f2d1002ff60 0x7f2d10032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:09.590 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.598+0000 7f2d2b7d2700 1 --2- 172.21.15.173:0/2582343870 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2d240f5410 0x7f2d24105630 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:09.590 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.598+0000 7f2d2b7d2700 1 -- 172.21.15.173:0/2582343870 >> 172.21.15.173:0/2582343870 conn(0x7f2d240f0b70 msgr2=0x7f2d240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:09.590 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.599+0000 7f2d2b7d2700 1 -- 172.21.15.173:0/2582343870 shutdown_connections 2024-01-21T19:57:09.590 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:09.599+0000 7f2d2b7d2700 1 -- 172.21.15.173:0/2582343870 wait complete. 2024-01-21T19:57:09.591 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:57:10.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:09 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:57:09.597776+0000 mon.smithi169 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.173:0/2582343870' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:57:10.970 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:57:10.970 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:57:11.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:10 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:57:09.016137+0000 mgr.smithi169.espgue (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:57:12.318 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.325+0000 7f44d0eb8700 1 -- 172.21.15.173:0/2853465707 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44cc0f5410 msgr2=0x7f44cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:12.318 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.325+0000 7f44d0eb8700 1 --2- 172.21.15.173:0/2853465707 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44cc0f5410 0x7f44cc0f5830 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f44bc009fb0 tx=0x7f44bc009b30).stop 2024-01-21T19:57:12.318 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.325+0000 7f44d0eb8700 1 -- 172.21.15.173:0/2853465707 shutdown_connections 2024-01-21T19:57:12.318 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.325+0000 7f44d0eb8700 1 --2- 172.21.15.173:0/2853465707 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44cc0f5410 0x7f44cc0f5830 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:12.318 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.325+0000 7f44d0eb8700 1 -- 172.21.15.173:0/2853465707 >> 172.21.15.173:0/2853465707 conn(0x7f44cc0f0b70 msgr2=0x7f44cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:12.319 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.325+0000 7f44d0eb8700 1 -- 172.21.15.173:0/2853465707 shutdown_connections 2024-01-21T19:57:12.319 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.325+0000 7f44d0eb8700 1 -- 172.21.15.173:0/2853465707 wait complete. 2024-01-21T19:57:12.319 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.326+0000 7f44d0eb8700 1 Processor -- start 2024-01-21T19:57:12.319 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.326+0000 7f44d0eb8700 1 -- start start 2024-01-21T19:57:12.319 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.326+0000 7f44d0eb8700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44cc0f5410 0x7f44cc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:12.319 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.326+0000 7f44d0eb8700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44cc105b20 con 0x7f44cc0f5410 2024-01-21T19:57:12.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.327+0000 7f44ca59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44cc0f5410 0x7f44cc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:12.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.327+0000 7f44ca59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44cc0f5410 0x7f44cc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:48568/0 (socket says 172.21.15.173:48568) 2024-01-21T19:57:12.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.327+0000 7f44ca59c700 1 -- 172.21.15.173:0/808702289 learned_addr learned my addr 172.21.15.173:0/808702289 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:57:12.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.327+0000 7f44ca59c700 1 -- 172.21.15.173:0/808702289 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44bc005040 con 0x7f44cc0f5410 2024-01-21T19:57:12.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.327+0000 7f44ca59c700 1 --2- 172.21.15.173:0/808702289 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44cc0f5410 0x7f44cc105630 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f44bc014000 tx=0x7f44bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:12.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.327+0000 7f44baffd700 1 -- 172.21.15.173:0/808702289 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44bc0095f0 con 0x7f44cc0f5410 2024-01-21T19:57:12.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.327+0000 7f44d0eb8700 1 -- 172.21.15.173:0/808702289 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44cc105d20 con 0x7f44cc0f5410 2024-01-21T19:57:12.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.327+0000 7f44baffd700 1 -- 172.21.15.173:0/808702289 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44bc007e70 con 0x7f44cc0f5410 2024-01-21T19:57:12.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.327+0000 7f44baffd700 1 -- 172.21.15.173:0/808702289 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44bc014ea0 con 0x7f44cc0f5410 2024-01-21T19:57:12.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.327+0000 7f44d0eb8700 1 -- 172.21.15.173:0/808702289 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44cc1061b0 con 0x7f44cc0f5410 2024-01-21T19:57:12.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.328+0000 7f44baffd700 1 -- 172.21.15.173:0/808702289 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44bc025070 con 0x7f44cc0f5410 2024-01-21T19:57:12.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.328+0000 7f44d0eb8700 1 -- 172.21.15.173:0/808702289 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44cc045ad0 con 0x7f44cc0f5410 2024-01-21T19:57:12.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.328+0000 7f44baffd700 1 --2- 172.21.15.173:0/808702289 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f44b402ff60 0x7f44b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:12.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.328+0000 7f44baffd700 1 -- 172.21.15.173:0/808702289 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f44bc040980 con 0x7f44cc0f5410 2024-01-21T19:57:12.322 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.329+0000 7f44c9d9b700 1 --2- 172.21.15.173:0/808702289 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f44b402ff60 0x7f44b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:12.322 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.329+0000 7f44c9d9b700 1 --2- 172.21.15.173:0/808702289 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f44b402ff60 0x7f44b4032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f44c00096a0 tx=0x7f44c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:12.325 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.333+0000 7f44baffd700 1 -- 172.21.15.173:0/808702289 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44bc0079c0 con 0x7f44cc0f5410 2024-01-21T19:57:12.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.541+0000 7f44d0eb8700 1 -- 172.21.15.173:0/808702289 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44cc0fae60 con 0x7f44cc0f5410 2024-01-21T19:57:12.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.541+0000 7f44baffd700 1 -- 172.21.15.173:0/808702289 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f44bc00f300 con 0x7f44cc0f5410 2024-01-21T19:57:12.534 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:57:12.534 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:57:12.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.543+0000 7f44d0eb8700 1 -- 172.21.15.173:0/808702289 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f44b402ff60 msgr2=0x7f44b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:12.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.543+0000 7f44d0eb8700 1 --2- 172.21.15.173:0/808702289 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f44b402ff60 0x7f44b4032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f44c00096a0 tx=0x7f44c0006b40).stop 2024-01-21T19:57:12.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.543+0000 7f44d0eb8700 1 -- 172.21.15.173:0/808702289 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44cc0f5410 msgr2=0x7f44cc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:12.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.543+0000 7f44d0eb8700 1 --2- 172.21.15.173:0/808702289 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44cc0f5410 0x7f44cc105630 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f44bc014000 tx=0x7f44bc00b040).stop 2024-01-21T19:57:12.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.543+0000 7f44d0eb8700 1 -- 172.21.15.173:0/808702289 shutdown_connections 2024-01-21T19:57:12.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.543+0000 7f44d0eb8700 1 --2- 172.21.15.173:0/808702289 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f44b402ff60 0x7f44b4032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:12.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.543+0000 7f44d0eb8700 1 --2- 172.21.15.173:0/808702289 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f44cc0f5410 0x7f44cc105630 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:12.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.543+0000 7f44d0eb8700 1 -- 172.21.15.173:0/808702289 >> 172.21.15.173:0/808702289 conn(0x7f44cc0f0b70 msgr2=0x7f44cc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:12.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.543+0000 7f44d0eb8700 1 -- 172.21.15.173:0/808702289 shutdown_connections 2024-01-21T19:57:12.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:12.543+0000 7f44d0eb8700 1 -- 172.21.15.173:0/808702289 wait complete. 2024-01-21T19:57:12.536 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:57:13.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:12 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:57:11.016438+0000 mgr.smithi169.espgue (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:57:13.176 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:12 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:57:12.543125+0000 mon.smithi169 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.173:0/808702289' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:57:13.944 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:57:13.944 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:57:15.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:14 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:57:13.016786+0000 mgr.smithi169.espgue (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:57:15.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.206+0000 7f5a5a68e700 1 -- 172.21.15.173:0/291322292 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a54060280 msgr2=0x7f5a540f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:15.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.206+0000 7f5a5a68e700 1 --2- 172.21.15.173:0/291322292 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a54060280 0x7f5a540f90f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f5a44009fb0 tx=0x7f5a44009b30).stop 2024-01-21T19:57:15.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.206+0000 7f5a5a68e700 1 -- 172.21.15.173:0/291322292 shutdown_connections 2024-01-21T19:57:15.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.206+0000 7f5a5a68e700 1 --2- 172.21.15.173:0/291322292 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a54060280 0x7f5a540f90f0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:15.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.206+0000 7f5a5a68e700 1 -- 172.21.15.173:0/291322292 >> 172.21.15.173:0/291322292 conn(0x7f5a540f0b70 msgr2=0x7f5a540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:15.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.206+0000 7f5a5a68e700 1 -- 172.21.15.173:0/291322292 shutdown_connections 2024-01-21T19:57:15.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.206+0000 7f5a5a68e700 1 -- 172.21.15.173:0/291322292 wait complete. 2024-01-21T19:57:15.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.206+0000 7f5a5a68e700 1 Processor -- start 2024-01-21T19:57:15.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.207+0000 7f5a5a68e700 1 -- start start 2024-01-21T19:57:15.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.207+0000 7f5a5a68e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a54060280 0x7f5a541066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:15.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.207+0000 7f5a5a68e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a54106bd0 con 0x7f5a54060280 2024-01-21T19:57:15.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.207+0000 7f5a53fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a54060280 0x7f5a541066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:15.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.207+0000 7f5a53fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a54060280 0x7f5a541066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:33012/0 (socket says 172.21.15.173:33012) 2024-01-21T19:57:15.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.207+0000 7f5a53fff700 1 -- 172.21.15.173:0/3288289119 learned_addr learned my addr 172.21.15.173:0/3288289119 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:57:15.201 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.207+0000 7f5a53fff700 1 -- 172.21.15.173:0/3288289119 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a44005040 con 0x7f5a54060280 2024-01-21T19:57:15.201 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.207+0000 7f5a53fff700 1 --2- 172.21.15.173:0/3288289119 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a54060280 0x7f5a541066e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f5a44013510 tx=0x7f5a4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:15.201 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.207+0000 7f5a50ff9700 1 -- 172.21.15.173:0/3288289119 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a44009320 con 0x7f5a54060280 2024-01-21T19:57:15.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.208+0000 7f5a50ff9700 1 -- 172.21.15.173:0/3288289119 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a44013f00 con 0x7f5a54060280 2024-01-21T19:57:15.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.208+0000 7f5a5a68e700 1 -- 172.21.15.173:0/3288289119 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a54106dd0 con 0x7f5a54060280 2024-01-21T19:57:15.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.208+0000 7f5a50ff9700 1 -- 172.21.15.173:0/3288289119 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a44014dc0 con 0x7f5a54060280 2024-01-21T19:57:15.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.208+0000 7f5a5a68e700 1 -- 172.21.15.173:0/3288289119 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a54107260 con 0x7f5a54060280 2024-01-21T19:57:15.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.208+0000 7f5a50ff9700 1 -- 172.21.15.173:0/3288289119 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5a440095f0 con 0x7f5a54060280 2024-01-21T19:57:15.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.208+0000 7f5a5a68e700 1 -- 172.21.15.173:0/3288289119 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a54045ad0 con 0x7f5a54060280 2024-01-21T19:57:15.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.209+0000 7f5a50ff9700 1 --2- 172.21.15.173:0/3288289119 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5a3c02ff60 0x7f5a3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:15.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.209+0000 7f5a50ff9700 1 -- 172.21.15.173:0/3288289119 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5a44041040 con 0x7f5a54060280 2024-01-21T19:57:15.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.209+0000 7f5a537fe700 1 --2- 172.21.15.173:0/3288289119 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5a3c02ff60 0x7f5a3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:15.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.209+0000 7f5a537fe700 1 --2- 172.21.15.173:0/3288289119 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5a3c02ff60 0x7f5a3c032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f5a480096a0 tx=0x7f5a48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:15.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.213+0000 7f5a50ff9700 1 -- 172.21.15.173:0/3288289119 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a44014420 con 0x7f5a54060280 2024-01-21T19:57:15.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.417+0000 7f5a5a68e700 1 -- 172.21.15.173:0/3288289119 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a54108110 con 0x7f5a54060280 2024-01-21T19:57:15.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.418+0000 7f5a50ff9700 1 -- 172.21.15.173:0/3288289119 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5a44010230 con 0x7f5a54060280 2024-01-21T19:57:15.410 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:57:15.410 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:57:15.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.419+0000 7f5a5a68e700 1 -- 172.21.15.173:0/3288289119 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5a3c02ff60 msgr2=0x7f5a3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:15.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.419+0000 7f5a5a68e700 1 --2- 172.21.15.173:0/3288289119 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5a3c02ff60 0x7f5a3c032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f5a480096a0 tx=0x7f5a48006b40).stop 2024-01-21T19:57:15.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.419+0000 7f5a5a68e700 1 -- 172.21.15.173:0/3288289119 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a54060280 msgr2=0x7f5a541066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:15.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.419+0000 7f5a5a68e700 1 --2- 172.21.15.173:0/3288289119 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a54060280 0x7f5a541066e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f5a44013510 tx=0x7f5a4400b040).stop 2024-01-21T19:57:15.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.420+0000 7f5a5a68e700 1 -- 172.21.15.173:0/3288289119 shutdown_connections 2024-01-21T19:57:15.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.420+0000 7f5a5a68e700 1 --2- 172.21.15.173:0/3288289119 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5a3c02ff60 0x7f5a3c032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:15.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.420+0000 7f5a5a68e700 1 --2- 172.21.15.173:0/3288289119 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5a54060280 0x7f5a541066e0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:15.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.420+0000 7f5a5a68e700 1 -- 172.21.15.173:0/3288289119 >> 172.21.15.173:0/3288289119 conn(0x7f5a540f0b70 msgr2=0x7f5a540f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:15.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.420+0000 7f5a5a68e700 1 -- 172.21.15.173:0/3288289119 shutdown_connections 2024-01-21T19:57:15.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:15.420+0000 7f5a5a68e700 1 -- 172.21.15.173:0/3288289119 wait complete. 2024-01-21T19:57:15.413 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:57:16.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:15 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:57:15.419689+0000 mon.smithi169 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.173:0/3288289119' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:57:16.812 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:57:16.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:57:17.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:16 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:57:15.017118+0000 mgr.smithi169.espgue (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:57:18.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.071+0000 7f9b622b0700 1 -- 172.21.15.173:0/1371750717 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b5c0f5410 msgr2=0x7f9b5c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:18.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.071+0000 7f9b622b0700 1 --2- 172.21.15.173:0/1371750717 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b5c0f5410 0x7f9b5c0f5830 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f9b4c009fb0 tx=0x7f9b4c009b30).stop 2024-01-21T19:57:18.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.071+0000 7f9b622b0700 1 -- 172.21.15.173:0/1371750717 shutdown_connections 2024-01-21T19:57:18.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.071+0000 7f9b622b0700 1 --2- 172.21.15.173:0/1371750717 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b5c0f5410 0x7f9b5c0f5830 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:18.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.071+0000 7f9b622b0700 1 -- 172.21.15.173:0/1371750717 >> 172.21.15.173:0/1371750717 conn(0x7f9b5c0f0b70 msgr2=0x7f9b5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:18.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.071+0000 7f9b622b0700 1 -- 172.21.15.173:0/1371750717 shutdown_connections 2024-01-21T19:57:18.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.071+0000 7f9b622b0700 1 -- 172.21.15.173:0/1371750717 wait complete. 2024-01-21T19:57:18.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.072+0000 7f9b622b0700 1 Processor -- start 2024-01-21T19:57:18.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.072+0000 7f9b622b0700 1 -- start start 2024-01-21T19:57:18.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.072+0000 7f9b622b0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b5c0f5410 0x7f9b5c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:18.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.072+0000 7f9b622b0700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c0fb820 con 0x7f9b5c0f5410 2024-01-21T19:57:18.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.073+0000 7f9b5bfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b5c0f5410 0x7f9b5c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:18.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.073+0000 7f9b5bfff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b5c0f5410 0x7f9b5c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:33026/0 (socket says 172.21.15.173:33026) 2024-01-21T19:57:18.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.073+0000 7f9b5bfff700 1 -- 172.21.15.173:0/1541139353 learned_addr learned my addr 172.21.15.173:0/1541139353 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:57:18.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.073+0000 7f9b5bfff700 1 -- 172.21.15.173:0/1541139353 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b4c005040 con 0x7f9b5c0f5410 2024-01-21T19:57:18.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.073+0000 7f9b5bfff700 1 --2- 172.21.15.173:0/1541139353 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b5c0f5410 0x7f9b5c0fd3a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f9b4c013510 tx=0x7f9b4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:18.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.073+0000 7f9b58ff9700 1 -- 172.21.15.173:0/1541139353 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b4c009320 con 0x7f9b5c0f5410 2024-01-21T19:57:18.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.073+0000 7f9b622b0700 1 -- 172.21.15.173:0/1541139353 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b5c0fb9c0 con 0x7f9b5c0f5410 2024-01-21T19:57:18.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.073+0000 7f9b58ff9700 1 -- 172.21.15.173:0/1541139353 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b4c013f00 con 0x7f9b5c0f5410 2024-01-21T19:57:18.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.073+0000 7f9b58ff9700 1 -- 172.21.15.173:0/1541139353 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b4c014d90 con 0x7f9b5c0f5410 2024-01-21T19:57:18.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.073+0000 7f9b622b0700 1 -- 172.21.15.173:0/1541139353 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b5c0fbe50 con 0x7f9b5c0f5410 2024-01-21T19:57:18.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.074+0000 7f9b58ff9700 1 -- 172.21.15.173:0/1541139353 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b4c014420 con 0x7f9b5c0f5410 2024-01-21T19:57:18.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.074+0000 7f9b622b0700 1 -- 172.21.15.173:0/1541139353 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b5c059100 con 0x7f9b5c0f5410 2024-01-21T19:57:18.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.074+0000 7f9b58ff9700 1 --2- 172.21.15.173:0/1541139353 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9b4402ff60 0x7f9b44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:18.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.074+0000 7f9b58ff9700 1 -- 172.21.15.173:0/1541139353 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b4c040ff0 con 0x7f9b5c0f5410 2024-01-21T19:57:18.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.074+0000 7f9b5b7fe700 1 --2- 172.21.15.173:0/1541139353 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9b4402ff60 0x7f9b44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:18.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.075+0000 7f9b5b7fe700 1 --2- 172.21.15.173:0/1541139353 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9b4402ff60 0x7f9b44032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f9b500096a0 tx=0x7f9b50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:18.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.079+0000 7f9b58ff9700 1 -- 172.21.15.173:0/1541139353 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b4c0095f0 con 0x7f9b5c0f5410 2024-01-21T19:57:18.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.282+0000 7f9b622b0700 1 -- 172.21.15.173:0/1541139353 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b5c0fae60 con 0x7f9b5c0f5410 2024-01-21T19:57:18.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.283+0000 7f9b58ff9700 1 -- 172.21.15.173:0/1541139353 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9b4c01db70 con 0x7f9b5c0f5410 2024-01-21T19:57:18.276 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:57:18.276 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:57:18.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.285+0000 7f9b622b0700 1 -- 172.21.15.173:0/1541139353 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9b4402ff60 msgr2=0x7f9b44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:18.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.285+0000 7f9b622b0700 1 --2- 172.21.15.173:0/1541139353 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9b4402ff60 0x7f9b44032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f9b500096a0 tx=0x7f9b50006b40).stop 2024-01-21T19:57:18.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.285+0000 7f9b622b0700 1 -- 172.21.15.173:0/1541139353 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b5c0f5410 msgr2=0x7f9b5c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:18.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.285+0000 7f9b622b0700 1 --2- 172.21.15.173:0/1541139353 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b5c0f5410 0x7f9b5c0fd3a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f9b4c013510 tx=0x7f9b4c00b040).stop 2024-01-21T19:57:18.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.285+0000 7f9b622b0700 1 -- 172.21.15.173:0/1541139353 shutdown_connections 2024-01-21T19:57:18.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.285+0000 7f9b622b0700 1 --2- 172.21.15.173:0/1541139353 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f9b4402ff60 0x7f9b44032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:18.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.285+0000 7f9b622b0700 1 --2- 172.21.15.173:0/1541139353 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9b5c0f5410 0x7f9b5c0fd3a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:18.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.285+0000 7f9b622b0700 1 -- 172.21.15.173:0/1541139353 >> 172.21.15.173:0/1541139353 conn(0x7f9b5c0f0b70 msgr2=0x7f9b5c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:18.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.285+0000 7f9b622b0700 1 -- 172.21.15.173:0/1541139353 shutdown_connections 2024-01-21T19:57:18.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:18.285+0000 7f9b622b0700 1 -- 172.21.15.173:0/1541139353 wait complete. 2024-01-21T19:57:18.278 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:57:19.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:18 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:57:17.017478+0000 mgr.smithi169.espgue (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:57:19.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:18 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:57:18.285031+0000 mon.smithi169 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.173:0/1541139353' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:57:19.667 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:57:19.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:57:20.818 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.824+0000 7fa8b3ca5700 1 -- 172.21.15.173:0/1349900955 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8ac0f6510 msgr2=0x7fa8ac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:20.818 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.824+0000 7fa8b3ca5700 1 --2- 172.21.15.173:0/1349900955 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8ac0f6510 0x7fa8ac0f6930 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fa89c009fb0 tx=0x7fa89c009b30).stop 2024-01-21T19:57:20.818 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.824+0000 7fa8b3ca5700 1 -- 172.21.15.173:0/1349900955 shutdown_connections 2024-01-21T19:57:20.818 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.824+0000 7fa8b3ca5700 1 --2- 172.21.15.173:0/1349900955 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8ac0f6510 0x7fa8ac0f6930 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:20.818 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.824+0000 7fa8b3ca5700 1 -- 172.21.15.173:0/1349900955 >> 172.21.15.173:0/1349900955 conn(0x7fa8ac0f1c70 msgr2=0x7fa8ac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:20.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.824+0000 7fa8b3ca5700 1 -- 172.21.15.173:0/1349900955 shutdown_connections 2024-01-21T19:57:20.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.824+0000 7fa8b3ca5700 1 -- 172.21.15.173:0/1349900955 wait complete. 2024-01-21T19:57:20.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.825+0000 7fa8b3ca5700 1 Processor -- start 2024-01-21T19:57:20.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.825+0000 7fa8b3ca5700 1 -- start start 2024-01-21T19:57:20.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.825+0000 7fa8b3ca5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8ac0f6510 0x7fa8ac06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:20.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.825+0000 7fa8b3ca5700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ac069e20 con 0x7fa8ac0f6510 2024-01-21T19:57:20.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.825+0000 7fa8b1a41700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8ac0f6510 0x7fa8ac06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:20.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.825+0000 7fa8b1a41700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8ac0f6510 0x7fa8ac06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:33038/0 (socket says 172.21.15.173:33038) 2024-01-21T19:57:20.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.825+0000 7fa8b1a41700 1 -- 172.21.15.173:0/3747188878 learned_addr learned my addr 172.21.15.173:0/3747188878 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:57:20.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.826+0000 7fa8b1a41700 1 -- 172.21.15.173:0/3747188878 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa89c005040 con 0x7fa8ac0f6510 2024-01-21T19:57:20.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.826+0000 7fa8b1a41700 1 --2- 172.21.15.173:0/3747188878 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8ac0f6510 0x7fa8ac06b9a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fa89c013510 tx=0x7fa89c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:20.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.826+0000 7fa8a27fc700 1 -- 172.21.15.173:0/3747188878 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa89c009320 con 0x7fa8ac0f6510 2024-01-21T19:57:20.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.826+0000 7fa8b3ca5700 1 -- 172.21.15.173:0/3747188878 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8ac069fc0 con 0x7fa8ac0f6510 2024-01-21T19:57:20.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.826+0000 7fa8a27fc700 1 -- 172.21.15.173:0/3747188878 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa89c013f00 con 0x7fa8ac0f6510 2024-01-21T19:57:20.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.826+0000 7fa8a27fc700 1 -- 172.21.15.173:0/3747188878 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa89c014d90 con 0x7fa8ac0f6510 2024-01-21T19:57:20.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.826+0000 7fa8b3ca5700 1 -- 172.21.15.173:0/3747188878 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8ac06a450 con 0x7fa8ac0f6510 2024-01-21T19:57:20.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.827+0000 7fa8a27fc700 1 -- 172.21.15.173:0/3747188878 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa89c014420 con 0x7fa8ac0f6510 2024-01-21T19:57:20.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.827+0000 7fa8b3ca5700 1 -- 172.21.15.173:0/3747188878 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8ac059100 con 0x7fa8ac0f6510 2024-01-21T19:57:20.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.827+0000 7fa8a27fc700 1 --2- 172.21.15.173:0/3747188878 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa89802ffb0 0x7fa898032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:20.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.827+0000 7fa8a27fc700 1 -- 172.21.15.173:0/3747188878 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa89c040ff0 con 0x7fa8ac0f6510 2024-01-21T19:57:20.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.827+0000 7fa8b1240700 1 --2- 172.21.15.173:0/3747188878 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa89802ffb0 0x7fa898032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:20.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.828+0000 7fa8b1240700 1 --2- 172.21.15.173:0/3747188878 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa89802ffb0 0x7fa898032470 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fa8a80096a0 tx=0x7fa8a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:20.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:20.832+0000 7fa8a27fc700 1 -- 172.21.15.173:0/3747188878 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa89c06d2b0 con 0x7fa8ac0f6510 2024-01-21T19:57:21.029 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:21.037+0000 7fa8b3ca5700 1 -- 172.21.15.173:0/3747188878 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8ac0fbf60 con 0x7fa8ac0f6510 2024-01-21T19:57:21.030 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:21.038+0000 7fa8a27fc700 1 -- 172.21.15.173:0/3747188878 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa89c0095f0 con 0x7fa8ac0f6510 2024-01-21T19:57:21.030 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:57:21.030 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:57:21.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:21.039+0000 7fa8b3ca5700 1 -- 172.21.15.173:0/3747188878 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa89802ffb0 msgr2=0x7fa898032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:21.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:21.039+0000 7fa8b3ca5700 1 --2- 172.21.15.173:0/3747188878 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa89802ffb0 0x7fa898032470 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fa8a80096a0 tx=0x7fa8a8006b40).stop 2024-01-21T19:57:21.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:21.039+0000 7fa8b3ca5700 1 -- 172.21.15.173:0/3747188878 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8ac0f6510 msgr2=0x7fa8ac06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:21.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:21.039+0000 7fa8b3ca5700 1 --2- 172.21.15.173:0/3747188878 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8ac0f6510 0x7fa8ac06b9a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fa89c013510 tx=0x7fa89c00b040).stop 2024-01-21T19:57:21.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:21.039+0000 7fa8b3ca5700 1 -- 172.21.15.173:0/3747188878 shutdown_connections 2024-01-21T19:57:21.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:21.039+0000 7fa8b3ca5700 1 --2- 172.21.15.173:0/3747188878 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fa89802ffb0 0x7fa898032470 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:21.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:21.039+0000 7fa8b3ca5700 1 --2- 172.21.15.173:0/3747188878 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa8ac0f6510 0x7fa8ac06b9a0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:21.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:21.039+0000 7fa8b3ca5700 1 -- 172.21.15.173:0/3747188878 >> 172.21.15.173:0/3747188878 conn(0x7fa8ac0f1c70 msgr2=0x7fa8ac0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:21.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:21.039+0000 7fa8b3ca5700 1 -- 172.21.15.173:0/3747188878 shutdown_connections 2024-01-21T19:57:21.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:21.040+0000 7fa8b3ca5700 1 -- 172.21.15.173:0/3747188878 wait complete. 2024-01-21T19:57:21.033 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:57:21.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:20 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:57:19.017835+0000 mgr.smithi169.espgue (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:57:22.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:21 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:57:21.039607+0000 mon.smithi169 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.173:0/3747188878' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:57:22.455 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:57:22.456 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:57:23.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:22 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:57:21.018204+0000 mgr.smithi169.espgue (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:57:23.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.652+0000 7febdc80c700 1 -- 172.21.15.173:0/207777178 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7febd80f5410 msgr2=0x7febd80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:23.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.652+0000 7febdc80c700 1 --2- 172.21.15.173:0/207777178 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7febd80f5410 0x7febd80f5830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7febc0009fb0 tx=0x7febc0009b30).stop 2024-01-21T19:57:23.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.652+0000 7febdc80c700 1 -- 172.21.15.173:0/207777178 shutdown_connections 2024-01-21T19:57:23.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.652+0000 7febdc80c700 1 --2- 172.21.15.173:0/207777178 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7febd80f5410 0x7febd80f5830 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:23.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.652+0000 7febdc80c700 1 -- 172.21.15.173:0/207777178 >> 172.21.15.173:0/207777178 conn(0x7febd80f0b70 msgr2=0x7febd80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:23.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.652+0000 7febdc80c700 1 -- 172.21.15.173:0/207777178 shutdown_connections 2024-01-21T19:57:23.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.652+0000 7febdc80c700 1 -- 172.21.15.173:0/207777178 wait complete. 2024-01-21T19:57:23.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.653+0000 7febdc80c700 1 Processor -- start 2024-01-21T19:57:23.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.653+0000 7febdc80c700 1 -- start start 2024-01-21T19:57:23.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.653+0000 7febdc80c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7febd80f5410 0x7febd810b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:23.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.653+0000 7febdc80c700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd8107240 con 0x7febd80f5410 2024-01-21T19:57:23.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.653+0000 7febd659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7febd80f5410 0x7febd810b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:23.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.653+0000 7febd659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7febd80f5410 0x7febd810b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:42640/0 (socket says 172.21.15.173:42640) 2024-01-21T19:57:23.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.653+0000 7febd659c700 1 -- 172.21.15.173:0/1513107501 learned_addr learned my addr 172.21.15.173:0/1513107501 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:57:23.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.653+0000 7febd659c700 1 -- 172.21.15.173:0/1513107501 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc0005040 con 0x7febd80f5410 2024-01-21T19:57:23.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.654+0000 7febd659c700 1 --2- 172.21.15.173:0/1513107501 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7febd80f5410 0x7febd810b850 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7febc0013510 tx=0x7febc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:23.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.654+0000 7febceffd700 1 -- 172.21.15.173:0/1513107501 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febc0009320 con 0x7febd80f5410 2024-01-21T19:57:23.648 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.654+0000 7febceffd700 1 -- 172.21.15.173:0/1513107501 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7febc0013f00 con 0x7febd80f5410 2024-01-21T19:57:23.648 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.654+0000 7febdc80c700 1 -- 172.21.15.173:0/1513107501 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd8060220 con 0x7febd80f5410 2024-01-21T19:57:23.648 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.654+0000 7febceffd700 1 -- 172.21.15.173:0/1513107501 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febc0014d90 con 0x7febd80f5410 2024-01-21T19:57:23.648 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.654+0000 7febdc80c700 1 -- 172.21.15.173:0/1513107501 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd810bd40 con 0x7febd80f5410 2024-01-21T19:57:23.649 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.655+0000 7febceffd700 1 -- 172.21.15.173:0/1513107501 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7febc00095f0 con 0x7febd80f5410 2024-01-21T19:57:23.649 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.655+0000 7febdc80c700 1 -- 172.21.15.173:0/1513107501 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd8045ad0 con 0x7febd80f5410 2024-01-21T19:57:23.649 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.655+0000 7febceffd700 1 --2- 172.21.15.173:0/1513107501 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7febc402ff60 0x7febc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:23.649 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.655+0000 7febceffd700 1 -- 172.21.15.173:0/1513107501 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7febc0041000 con 0x7febd80f5410 2024-01-21T19:57:23.649 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.655+0000 7febd5d9b700 1 --2- 172.21.15.173:0/1513107501 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7febc402ff60 0x7febc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:23.649 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.656+0000 7febd5d9b700 1 --2- 172.21.15.173:0/1513107501 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7febc402ff60 0x7febc4032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7febc80096a0 tx=0x7febc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:23.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.660+0000 7febceffd700 1 -- 172.21.15.173:0/1513107501 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7febc0014420 con 0x7febd80f5410 2024-01-21T19:57:23.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.863+0000 7febdc80c700 1 -- 172.21.15.173:0/1513107501 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7febd80fae60 con 0x7febd80f5410 2024-01-21T19:57:23.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.864+0000 7febceffd700 1 -- 172.21.15.173:0/1513107501 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7febc001d7f0 con 0x7febd80f5410 2024-01-21T19:57:23.857 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:57:23.857 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:57:23.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.866+0000 7febdc80c700 1 -- 172.21.15.173:0/1513107501 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7febc402ff60 msgr2=0x7febc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:23.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.866+0000 7febdc80c700 1 --2- 172.21.15.173:0/1513107501 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7febc402ff60 0x7febc4032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7febc80096a0 tx=0x7febc8006b40).stop 2024-01-21T19:57:23.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.866+0000 7febdc80c700 1 -- 172.21.15.173:0/1513107501 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7febd80f5410 msgr2=0x7febd810b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:23.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.866+0000 7febdc80c700 1 --2- 172.21.15.173:0/1513107501 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7febd80f5410 0x7febd810b850 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7febc0013510 tx=0x7febc000b040).stop 2024-01-21T19:57:23.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.866+0000 7febdc80c700 1 -- 172.21.15.173:0/1513107501 shutdown_connections 2024-01-21T19:57:23.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.866+0000 7febdc80c700 1 --2- 172.21.15.173:0/1513107501 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7febc402ff60 0x7febc4032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:23.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.866+0000 7febdc80c700 1 --2- 172.21.15.173:0/1513107501 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7febd80f5410 0x7febd810b850 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:23.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.866+0000 7febdc80c700 1 -- 172.21.15.173:0/1513107501 >> 172.21.15.173:0/1513107501 conn(0x7febd80f0b70 msgr2=0x7febd80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:23.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.866+0000 7febdc80c700 1 -- 172.21.15.173:0/1513107501 shutdown_connections 2024-01-21T19:57:23.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:23.866+0000 7febdc80c700 1 -- 172.21.15.173:0/1513107501 wait complete. 2024-01-21T19:57:23.859 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:57:25.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:24 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:57:23.018569+0000 mgr.smithi169.espgue (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:57:25.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:24 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:57:23.866008+0000 mon.smithi169 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.173:0/1513107501' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:57:25.317 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:57:25.317 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:57:26.483 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.491+0000 7fe554dd8700 1 -- 172.21.15.173:0/2747185571 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5500f6510 msgr2=0x7fe5500f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:26.484 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.491+0000 7fe554dd8700 1 --2- 172.21.15.173:0/2747185571 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5500f6510 0x7fe5500f6930 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fe538009fb0 tx=0x7fe538009b30).stop 2024-01-21T19:57:26.484 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.491+0000 7fe554dd8700 1 -- 172.21.15.173:0/2747185571 shutdown_connections 2024-01-21T19:57:26.484 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.491+0000 7fe554dd8700 1 --2- 172.21.15.173:0/2747185571 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5500f6510 0x7fe5500f6930 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:26.484 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.491+0000 7fe554dd8700 1 -- 172.21.15.173:0/2747185571 >> 172.21.15.173:0/2747185571 conn(0x7fe5500f1c70 msgr2=0x7fe5500f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:26.484 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.491+0000 7fe554dd8700 1 -- 172.21.15.173:0/2747185571 shutdown_connections 2024-01-21T19:57:26.484 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.491+0000 7fe554dd8700 1 -- 172.21.15.173:0/2747185571 wait complete. 2024-01-21T19:57:26.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.492+0000 7fe554dd8700 1 Processor -- start 2024-01-21T19:57:26.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.492+0000 7fe554dd8700 1 -- start start 2024-01-21T19:57:26.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.492+0000 7fe554dd8700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5500f6510 0x7fe550106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:26.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.492+0000 7fe554dd8700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe550106c20 con 0x7fe5500f6510 2024-01-21T19:57:26.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.492+0000 7fe54e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5500f6510 0x7fe550106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:26.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.492+0000 7fe54e59c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5500f6510 0x7fe550106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:42654/0 (socket says 172.21.15.173:42654) 2024-01-21T19:57:26.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.492+0000 7fe54e59c700 1 -- 172.21.15.173:0/3135667058 learned_addr learned my addr 172.21.15.173:0/3135667058 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:57:26.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.492+0000 7fe54e59c700 1 -- 172.21.15.173:0/3135667058 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe538005040 con 0x7fe5500f6510 2024-01-21T19:57:26.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.492+0000 7fe54e59c700 1 --2- 172.21.15.173:0/3135667058 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5500f6510 0x7fe550106730 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fe538013510 tx=0x7fe53800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:26.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.493+0000 7fe546ffd700 1 -- 172.21.15.173:0/3135667058 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe538009320 con 0x7fe5500f6510 2024-01-21T19:57:26.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.493+0000 7fe554dd8700 1 -- 172.21.15.173:0/3135667058 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe550106e20 con 0x7fe5500f6510 2024-01-21T19:57:26.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.493+0000 7fe546ffd700 1 -- 172.21.15.173:0/3135667058 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe538013f00 con 0x7fe5500f6510 2024-01-21T19:57:26.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.493+0000 7fe546ffd700 1 -- 172.21.15.173:0/3135667058 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe538014d90 con 0x7fe5500f6510 2024-01-21T19:57:26.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.493+0000 7fe554dd8700 1 -- 172.21.15.173:0/3135667058 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5501072b0 con 0x7fe5500f6510 2024-01-21T19:57:26.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.493+0000 7fe546ffd700 1 -- 172.21.15.173:0/3135667058 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe5380095f0 con 0x7fe5500f6510 2024-01-21T19:57:26.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.493+0000 7fe554dd8700 1 -- 172.21.15.173:0/3135667058 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe550045ad0 con 0x7fe5500f6510 2024-01-21T19:57:26.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.494+0000 7fe546ffd700 1 --2- 172.21.15.173:0/3135667058 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe53c02ff60 0x7fe53c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:26.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.494+0000 7fe546ffd700 1 -- 172.21.15.173:0/3135667058 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe538041000 con 0x7fe5500f6510 2024-01-21T19:57:26.488 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.494+0000 7fe54dd9b700 1 --2- 172.21.15.173:0/3135667058 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe53c02ff60 0x7fe53c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:26.488 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.494+0000 7fe54dd9b700 1 --2- 172.21.15.173:0/3135667058 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe53c02ff60 0x7fe53c032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fe5400096a0 tx=0x7fe540006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:26.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.499+0000 7fe546ffd700 1 -- 172.21.15.173:0/3135667058 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe538014420 con 0x7fe5500f6510 2024-01-21T19:57:26.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.705+0000 7fe554dd8700 1 -- 172.21.15.173:0/3135667058 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe5500fbf60 con 0x7fe5500f6510 2024-01-21T19:57:26.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.706+0000 7fe546ffd700 1 -- 172.21.15.173:0/3135667058 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe53801d7f0 con 0x7fe5500f6510 2024-01-21T19:57:26.698 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:57:26.698 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:57:26.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.707+0000 7fe554dd8700 1 -- 172.21.15.173:0/3135667058 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe53c02ff60 msgr2=0x7fe53c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:26.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.707+0000 7fe554dd8700 1 --2- 172.21.15.173:0/3135667058 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe53c02ff60 0x7fe53c032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fe5400096a0 tx=0x7fe540006b40).stop 2024-01-21T19:57:26.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.707+0000 7fe554dd8700 1 -- 172.21.15.173:0/3135667058 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5500f6510 msgr2=0x7fe550106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:26.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.707+0000 7fe554dd8700 1 --2- 172.21.15.173:0/3135667058 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5500f6510 0x7fe550106730 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fe538013510 tx=0x7fe53800b040).stop 2024-01-21T19:57:26.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.707+0000 7fe554dd8700 1 -- 172.21.15.173:0/3135667058 shutdown_connections 2024-01-21T19:57:26.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.707+0000 7fe554dd8700 1 --2- 172.21.15.173:0/3135667058 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fe53c02ff60 0x7fe53c032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:26.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.707+0000 7fe554dd8700 1 --2- 172.21.15.173:0/3135667058 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5500f6510 0x7fe550106730 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:26.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.707+0000 7fe554dd8700 1 -- 172.21.15.173:0/3135667058 >> 172.21.15.173:0/3135667058 conn(0x7fe5500f1c70 msgr2=0x7fe5500f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:26.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.707+0000 7fe554dd8700 1 -- 172.21.15.173:0/3135667058 shutdown_connections 2024-01-21T19:57:26.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:26.707+0000 7fe554dd8700 1 -- 172.21.15.173:0/3135667058 wait complete. 2024-01-21T19:57:26.700 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:57:27.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:26 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:57:25.018890+0000 mgr.smithi169.espgue (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:57:27.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:26 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:57:26.707447+0000 mon.smithi169 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.173:0/3135667058' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:57:28.077 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:57:28.078 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:57:29.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:28 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:57:27.019246+0000 mgr.smithi169.espgue (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:57:29.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.239+0000 7f4baab6d700 1 -- 172.21.15.173:0/1022776833 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ba40f78e0 msgr2=0x7f4ba40f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:29.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.239+0000 7f4baab6d700 1 --2- 172.21.15.173:0/1022776833 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ba40f78e0 0x7f4ba40f7d00 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f4b94009fb0 tx=0x7f4b94009b30).stop 2024-01-21T19:57:29.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.239+0000 7f4baab6d700 1 -- 172.21.15.173:0/1022776833 shutdown_connections 2024-01-21T19:57:29.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.239+0000 7f4baab6d700 1 --2- 172.21.15.173:0/1022776833 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ba40f78e0 0x7f4ba40f7d00 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:29.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.239+0000 7f4baab6d700 1 -- 172.21.15.173:0/1022776833 >> 172.21.15.173:0/1022776833 conn(0x7f4ba40f0120 msgr2=0x7f4ba40f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:29.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.239+0000 7f4baab6d700 1 -- 172.21.15.173:0/1022776833 shutdown_connections 2024-01-21T19:57:29.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.239+0000 7f4baab6d700 1 -- 172.21.15.173:0/1022776833 wait complete. 2024-01-21T19:57:29.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.239+0000 7f4baab6d700 1 Processor -- start 2024-01-21T19:57:29.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.240+0000 7f4baab6d700 1 -- start start 2024-01-21T19:57:29.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.240+0000 7f4baab6d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ba40f78e0 0x7f4ba41064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:29.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.240+0000 7f4baab6d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba41069d0 con 0x7f4ba40f78e0 2024-01-21T19:57:29.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.240+0000 7f4ba9b6b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ba40f78e0 0x7f4ba41064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:29.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.240+0000 7f4ba9b6b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ba40f78e0 0x7f4ba41064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:42680/0 (socket says 172.21.15.173:42680) 2024-01-21T19:57:29.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.240+0000 7f4ba9b6b700 1 -- 172.21.15.173:0/3981662721 learned_addr learned my addr 172.21.15.173:0/3981662721 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:57:29.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.240+0000 7f4ba9b6b700 1 -- 172.21.15.173:0/3981662721 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b94005040 con 0x7f4ba40f78e0 2024-01-21T19:57:29.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.240+0000 7f4ba9b6b700 1 --2- 172.21.15.173:0/3981662721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ba40f78e0 0x7f4ba41064e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f4b94013510 tx=0x7f4b9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:29.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.240+0000 7f4b9a7fc700 1 -- 172.21.15.173:0/3981662721 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b94009320 con 0x7f4ba40f78e0 2024-01-21T19:57:29.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.241+0000 7f4baab6d700 1 -- 172.21.15.173:0/3981662721 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba4106bd0 con 0x7f4ba40f78e0 2024-01-21T19:57:29.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.241+0000 7f4b9a7fc700 1 -- 172.21.15.173:0/3981662721 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b94013f00 con 0x7f4ba40f78e0 2024-01-21T19:57:29.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.241+0000 7f4b9a7fc700 1 -- 172.21.15.173:0/3981662721 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b94014dc0 con 0x7f4ba40f78e0 2024-01-21T19:57:29.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.241+0000 7f4baab6d700 1 -- 172.21.15.173:0/3981662721 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba4107060 con 0x7f4ba40f78e0 2024-01-21T19:57:29.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.241+0000 7f4baab6d700 1 -- 172.21.15.173:0/3981662721 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba4045ad0 con 0x7f4ba40f78e0 2024-01-21T19:57:29.235 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.242+0000 7f4b9a7fc700 1 -- 172.21.15.173:0/3981662721 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b940095f0 con 0x7f4ba40f78e0 2024-01-21T19:57:29.235 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.242+0000 7f4b9a7fc700 1 --2- 172.21.15.173:0/3981662721 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4b9002ff60 0x7f4b90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:29.235 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.242+0000 7f4b9a7fc700 1 -- 172.21.15.173:0/3981662721 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b94041000 con 0x7f4ba40f78e0 2024-01-21T19:57:29.235 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.242+0000 7f4ba936a700 1 --2- 172.21.15.173:0/3981662721 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4b9002ff60 0x7f4b90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:29.235 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.242+0000 7f4ba936a700 1 --2- 172.21.15.173:0/3981662721 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4b9002ff60 0x7f4b90032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f4ba00096a0 tx=0x7f4ba0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:29.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.247+0000 7f4b9a7fc700 1 -- 172.21.15.173:0/3981662721 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b9401c790 con 0x7f4ba40f78e0 2024-01-21T19:57:29.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.453+0000 7f4baab6d700 1 -- 172.21.15.173:0/3981662721 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ba4107f10 con 0x7f4ba40f78e0 2024-01-21T19:57:29.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.454+0000 7f4b9a7fc700 1 -- 172.21.15.173:0/3981662721 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4b94010230 con 0x7f4ba40f78e0 2024-01-21T19:57:29.446 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:57:29.446 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:57:29.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.455+0000 7f4baab6d700 1 -- 172.21.15.173:0/3981662721 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4b9002ff60 msgr2=0x7f4b90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:29.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.455+0000 7f4baab6d700 1 --2- 172.21.15.173:0/3981662721 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4b9002ff60 0x7f4b90032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f4ba00096a0 tx=0x7f4ba0006b40).stop 2024-01-21T19:57:29.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.455+0000 7f4baab6d700 1 -- 172.21.15.173:0/3981662721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ba40f78e0 msgr2=0x7f4ba41064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:29.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.455+0000 7f4baab6d700 1 --2- 172.21.15.173:0/3981662721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ba40f78e0 0x7f4ba41064e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f4b94013510 tx=0x7f4b9400b040).stop 2024-01-21T19:57:29.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.455+0000 7f4baab6d700 1 -- 172.21.15.173:0/3981662721 shutdown_connections 2024-01-21T19:57:29.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.455+0000 7f4baab6d700 1 --2- 172.21.15.173:0/3981662721 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f4b9002ff60 0x7f4b90032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:29.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.455+0000 7f4baab6d700 1 --2- 172.21.15.173:0/3981662721 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4ba40f78e0 0x7f4ba41064e0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:29.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.455+0000 7f4baab6d700 1 -- 172.21.15.173:0/3981662721 >> 172.21.15.173:0/3981662721 conn(0x7f4ba40f0120 msgr2=0x7f4ba40f0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:29.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.455+0000 7f4baab6d700 1 -- 172.21.15.173:0/3981662721 shutdown_connections 2024-01-21T19:57:29.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:29.455+0000 7f4baab6d700 1 -- 172.21.15.173:0/3981662721 wait complete. 2024-01-21T19:57:29.449 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:57:30.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:29 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:57:29.455530+0000 mon.smithi169 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.173:0/3981662721' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:57:30.849 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:57:30.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:57:31.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:30 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:57:29.019621+0000 mgr.smithi169.espgue (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:57:32.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.014+0000 7f62cddbd700 1 -- 172.21.15.173:0/1834466643 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c80f4800 msgr2=0x7f62c80f4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:32.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.014+0000 7f62cddbd700 1 --2- 172.21.15.173:0/1834466643 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c80f4800 0x7f62c80f4c20 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f62bc009fb0 tx=0x7f62bc009b30).stop 2024-01-21T19:57:32.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.014+0000 7f62cddbd700 1 -- 172.21.15.173:0/1834466643 shutdown_connections 2024-01-21T19:57:32.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.014+0000 7f62cddbd700 1 --2- 172.21.15.173:0/1834466643 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c80f4800 0x7f62c80f4c20 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:32.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.014+0000 7f62cddbd700 1 -- 172.21.15.173:0/1834466643 >> 172.21.15.173:0/1834466643 conn(0x7f62c80f0260 msgr2=0x7f62c80f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:32.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.014+0000 7f62cddbd700 1 -- 172.21.15.173:0/1834466643 shutdown_connections 2024-01-21T19:57:32.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.014+0000 7f62cddbd700 1 -- 172.21.15.173:0/1834466643 wait complete. 2024-01-21T19:57:32.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.015+0000 7f62cddbd700 1 Processor -- start 2024-01-21T19:57:32.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.015+0000 7f62cddbd700 1 -- start start 2024-01-21T19:57:32.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.015+0000 7f62cddbd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c80f4800 0x7f62c81066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:32.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.015+0000 7f62cddbd700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c8106ba0 con 0x7f62c80f4800 2024-01-21T19:57:32.008 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.015+0000 7f62c77fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c80f4800 0x7f62c81066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:32.008 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.015+0000 7f62c77fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c80f4800 0x7f62c81066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:42684/0 (socket says 172.21.15.173:42684) 2024-01-21T19:57:32.008 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.015+0000 7f62c77fe700 1 -- 172.21.15.173:0/609891321 learned_addr learned my addr 172.21.15.173:0/609891321 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:57:32.008 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.016+0000 7f62c77fe700 1 -- 172.21.15.173:0/609891321 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62bc005040 con 0x7f62c80f4800 2024-01-21T19:57:32.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.016+0000 7f62c77fe700 1 --2- 172.21.15.173:0/609891321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c80f4800 0x7f62c81066b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f62bc013510 tx=0x7f62bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:32.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.016+0000 7f62affff700 1 -- 172.21.15.173:0/609891321 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62bc009320 con 0x7f62c80f4800 2024-01-21T19:57:32.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.016+0000 7f62cddbd700 1 -- 172.21.15.173:0/609891321 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c8106da0 con 0x7f62c80f4800 2024-01-21T19:57:32.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.016+0000 7f62affff700 1 -- 172.21.15.173:0/609891321 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62bc013f00 con 0x7f62c80f4800 2024-01-21T19:57:32.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.016+0000 7f62affff700 1 -- 172.21.15.173:0/609891321 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62bc014dc0 con 0x7f62c80f4800 2024-01-21T19:57:32.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.016+0000 7f62cddbd700 1 -- 172.21.15.173:0/609891321 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c8107230 con 0x7f62c80f4800 2024-01-21T19:57:32.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.017+0000 7f62affff700 1 -- 172.21.15.173:0/609891321 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f62bc0095f0 con 0x7f62c80f4800 2024-01-21T19:57:32.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.017+0000 7f62cddbd700 1 -- 172.21.15.173:0/609891321 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c8045ad0 con 0x7f62c80f4800 2024-01-21T19:57:32.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.017+0000 7f62affff700 1 --2- 172.21.15.173:0/609891321 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f62b002ff60 0x7f62b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:32.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.017+0000 7f62affff700 1 -- 172.21.15.173:0/609891321 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f62bc041030 con 0x7f62c80f4800 2024-01-21T19:57:32.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.017+0000 7f62c6ffd700 1 --2- 172.21.15.173:0/609891321 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f62b002ff60 0x7f62b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:32.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.018+0000 7f62c6ffd700 1 --2- 172.21.15.173:0/609891321 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f62b002ff60 0x7f62b0032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f62b80096a0 tx=0x7f62b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:32.013 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.022+0000 7f62affff700 1 -- 172.21.15.173:0/609891321 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62bc014420 con 0x7f62c80f4800 2024-01-21T19:57:32.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:57:31.078039+0000 mon.smithi169 (mon.0) 323 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/mirror_snapshot_schedule"}]: dispatch 2024-01-21T19:57:32.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:31 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:57:31.080924+0000 mon.smithi169 (mon.0) 324 : audit [INF] from='mgr.14162 172.21.15.169:0/618373488' entity='mgr.smithi169.espgue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.espgue/trash_purge_schedule"}]: dispatch 2024-01-21T19:57:32.221 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.230+0000 7f62cddbd700 1 -- 172.21.15.173:0/609891321 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f62c81080e0 con 0x7f62c80f4800 2024-01-21T19:57:32.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.231+0000 7f62affff700 1 -- 172.21.15.173:0/609891321 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f62bc010230 con 0x7f62c80f4800 2024-01-21T19:57:32.222 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:57:32.222 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:57:32.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.233+0000 7f62cddbd700 1 -- 172.21.15.173:0/609891321 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f62b002ff60 msgr2=0x7f62b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:32.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.233+0000 7f62cddbd700 1 --2- 172.21.15.173:0/609891321 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f62b002ff60 0x7f62b0032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f62b80096a0 tx=0x7f62b8006b40).stop 2024-01-21T19:57:32.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.233+0000 7f62cddbd700 1 -- 172.21.15.173:0/609891321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c80f4800 msgr2=0x7f62c81066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:32.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.233+0000 7f62cddbd700 1 --2- 172.21.15.173:0/609891321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c80f4800 0x7f62c81066b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f62bc013510 tx=0x7f62bc00b040).stop 2024-01-21T19:57:32.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.233+0000 7f62cddbd700 1 -- 172.21.15.173:0/609891321 shutdown_connections 2024-01-21T19:57:32.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.233+0000 7f62cddbd700 1 --2- 172.21.15.173:0/609891321 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f62b002ff60 0x7f62b0032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:32.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.233+0000 7f62cddbd700 1 --2- 172.21.15.173:0/609891321 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c80f4800 0x7f62c81066b0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:32.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.233+0000 7f62cddbd700 1 -- 172.21.15.173:0/609891321 >> 172.21.15.173:0/609891321 conn(0x7f62c80f0260 msgr2=0x7f62c80f0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:32.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.233+0000 7f62cddbd700 1 -- 172.21.15.173:0/609891321 shutdown_connections 2024-01-21T19:57:32.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:32.233+0000 7f62cddbd700 1 -- 172.21.15.173:0/609891321 wait complete. 2024-01-21T19:57:32.225 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:57:33.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:32 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:57:31.019876+0000 mgr.smithi169.espgue (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:57:33.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:32 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:57:32.232008+0000 mon.smithi169 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.173:0/609891321' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:57:33.610 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:57:33.610 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:57:34.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.814+0000 7f5ca7780700 1 -- 172.21.15.173:0/2131892664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ca00f5410 msgr2=0x7f5ca00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:34.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.814+0000 7f5ca7780700 1 --2- 172.21.15.173:0/2131892664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ca00f5410 0x7f5ca00f5830 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f5c90009fb0 tx=0x7f5c90009b30).stop 2024-01-21T19:57:34.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.814+0000 7f5ca7780700 1 -- 172.21.15.173:0/2131892664 shutdown_connections 2024-01-21T19:57:34.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.814+0000 7f5ca7780700 1 --2- 172.21.15.173:0/2131892664 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ca00f5410 0x7f5ca00f5830 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:34.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.814+0000 7f5ca7780700 1 -- 172.21.15.173:0/2131892664 >> 172.21.15.173:0/2131892664 conn(0x7f5ca00f0b70 msgr2=0x7f5ca00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:34.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.814+0000 7f5ca7780700 1 -- 172.21.15.173:0/2131892664 shutdown_connections 2024-01-21T19:57:34.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.815+0000 7f5ca7780700 1 -- 172.21.15.173:0/2131892664 wait complete. 2024-01-21T19:57:34.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.815+0000 7f5ca7780700 1 Processor -- start 2024-01-21T19:57:34.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.815+0000 7f5ca7780700 1 -- start start 2024-01-21T19:57:34.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.815+0000 7f5ca7780700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ca00f5410 0x7f5ca0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:34.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.815+0000 7f5ca7780700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca0105b20 con 0x7f5ca00f5410 2024-01-21T19:57:34.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.816+0000 7f5ca551c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ca00f5410 0x7f5ca0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:34.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.816+0000 7f5ca551c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ca00f5410 0x7f5ca0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:56112/0 (socket says 172.21.15.173:56112) 2024-01-21T19:57:34.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.816+0000 7f5ca551c700 1 -- 172.21.15.173:0/4247100881 learned_addr learned my addr 172.21.15.173:0/4247100881 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:57:34.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.816+0000 7f5ca551c700 1 -- 172.21.15.173:0/4247100881 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c90005040 con 0x7f5ca00f5410 2024-01-21T19:57:34.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.816+0000 7f5ca551c700 1 --2- 172.21.15.173:0/4247100881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ca00f5410 0x7f5ca0105630 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f5c90013510 tx=0x7f5c9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:34.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.816+0000 7f5c95ffb700 1 -- 172.21.15.173:0/4247100881 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c90009320 con 0x7f5ca00f5410 2024-01-21T19:57:34.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.816+0000 7f5ca7780700 1 -- 172.21.15.173:0/4247100881 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca0105d20 con 0x7f5ca00f5410 2024-01-21T19:57:34.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.816+0000 7f5c95ffb700 1 -- 172.21.15.173:0/4247100881 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c90013f00 con 0x7f5ca00f5410 2024-01-21T19:57:34.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.816+0000 7f5c95ffb700 1 -- 172.21.15.173:0/4247100881 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c90014d90 con 0x7f5ca00f5410 2024-01-21T19:57:34.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.816+0000 7f5ca7780700 1 -- 172.21.15.173:0/4247100881 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca01061b0 con 0x7f5ca00f5410 2024-01-21T19:57:34.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.817+0000 7f5c95ffb700 1 -- 172.21.15.173:0/4247100881 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5c90014420 con 0x7f5ca00f5410 2024-01-21T19:57:34.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.817+0000 7f5ca7780700 1 -- 172.21.15.173:0/4247100881 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca0045ad0 con 0x7f5ca00f5410 2024-01-21T19:57:34.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.817+0000 7f5c95ffb700 1 --2- 172.21.15.173:0/4247100881 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5c8c02ff60 0x7f5c8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:34.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.817+0000 7f5c95ffb700 1 -- 172.21.15.173:0/4247100881 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5c90041080 con 0x7f5ca00f5410 2024-01-21T19:57:34.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.818+0000 7f5ca4d1b700 1 --2- 172.21.15.173:0/4247100881 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5c8c02ff60 0x7f5c8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:34.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.818+0000 7f5ca4d1b700 1 --2- 172.21.15.173:0/4247100881 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5c8c02ff60 0x7f5c8c032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f5c9c0096a0 tx=0x7f5c9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:34.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:34.822+0000 7f5c95ffb700 1 -- 172.21.15.173:0/4247100881 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c900095f0 con 0x7f5ca00f5410 2024-01-21T19:57:35.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:35.027+0000 7f5ca7780700 1 -- 172.21.15.173:0/4247100881 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ca00fae60 con 0x7f5ca00f5410 2024-01-21T19:57:35.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:35.028+0000 7f5c95ffb700 1 -- 172.21.15.173:0/4247100881 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5c9001db70 con 0x7f5ca00f5410 2024-01-21T19:57:35.020 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:57:35.020 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:57:35.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:35.030+0000 7f5ca7780700 1 -- 172.21.15.173:0/4247100881 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5c8c02ff60 msgr2=0x7f5c8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:35.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:35.030+0000 7f5ca7780700 1 --2- 172.21.15.173:0/4247100881 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5c8c02ff60 0x7f5c8c032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f5c9c0096a0 tx=0x7f5c9c006b40).stop 2024-01-21T19:57:35.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:35.030+0000 7f5ca7780700 1 -- 172.21.15.173:0/4247100881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ca00f5410 msgr2=0x7f5ca0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:35.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:35.030+0000 7f5ca7780700 1 --2- 172.21.15.173:0/4247100881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ca00f5410 0x7f5ca0105630 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f5c90013510 tx=0x7f5c9000b040).stop 2024-01-21T19:57:35.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:35.030+0000 7f5ca7780700 1 -- 172.21.15.173:0/4247100881 shutdown_connections 2024-01-21T19:57:35.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:35.030+0000 7f5ca7780700 1 --2- 172.21.15.173:0/4247100881 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7f5c8c02ff60 0x7f5c8c032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:35.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:35.030+0000 7f5ca7780700 1 --2- 172.21.15.173:0/4247100881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5ca00f5410 0x7f5ca0105630 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:35.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:35.030+0000 7f5ca7780700 1 -- 172.21.15.173:0/4247100881 >> 172.21.15.173:0/4247100881 conn(0x7f5ca00f0b70 msgr2=0x7f5ca00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:35.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:35.030+0000 7f5ca7780700 1 -- 172.21.15.173:0/4247100881 shutdown_connections 2024-01-21T19:57:35.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:35.030+0000 7f5ca7780700 1 -- 172.21.15.173:0/4247100881 wait complete. 2024-01-21T19:57:35.022 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:57:35.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:34 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:57:33.020114+0000 mgr.smithi169.espgue (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:57:36.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:35 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:57:35.029110+0000 mon.smithi169 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.173:0/4247100881' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:57:36.416 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:57:36.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:57:37.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:36 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:57:35.020396+0000 mgr.smithi169.espgue (mgr.14162) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:57:37.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.680+0000 7fcabb52c700 1 -- 172.21.15.173:0/1049361269 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcab40f62a0 msgr2=0x7fcab40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:37.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.680+0000 7fcabb52c700 1 --2- 172.21.15.173:0/1049361269 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcab40f62a0 0x7fcab40f66c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7fcab0009fb0 tx=0x7fcab0009b30).stop 2024-01-21T19:57:37.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.681+0000 7fcabb52c700 1 -- 172.21.15.173:0/1049361269 shutdown_connections 2024-01-21T19:57:37.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.681+0000 7fcabb52c700 1 --2- 172.21.15.173:0/1049361269 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcab40f62a0 0x7fcab40f66c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:37.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.681+0000 7fcabb52c700 1 -- 172.21.15.173:0/1049361269 >> 172.21.15.173:0/1049361269 conn(0x7fcab40f1aa0 msgr2=0x7fcab40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:37.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.681+0000 7fcabb52c700 1 -- 172.21.15.173:0/1049361269 shutdown_connections 2024-01-21T19:57:37.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.681+0000 7fcabb52c700 1 -- 172.21.15.173:0/1049361269 wait complete. 2024-01-21T19:57:37.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.681+0000 7fcabb52c700 1 Processor -- start 2024-01-21T19:57:37.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.681+0000 7fcabb52c700 1 -- start start 2024-01-21T19:57:37.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.682+0000 7fcabb52c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcab40f62a0 0x7fcab41086b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:37.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.682+0000 7fcabb52c700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab4108ba0 con 0x7fcab40f62a0 2024-01-21T19:57:37.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.682+0000 7fcaba52a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcab40f62a0 0x7fcab41086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:37.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.682+0000 7fcaba52a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcab40f62a0 0x7fcab41086b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:56126/0 (socket says 172.21.15.173:56126) 2024-01-21T19:57:37.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.682+0000 7fcaba52a700 1 -- 172.21.15.173:0/537994679 learned_addr learned my addr 172.21.15.173:0/537994679 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:57:37.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.682+0000 7fcaba52a700 1 -- 172.21.15.173:0/537994679 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab0005040 con 0x7fcab40f62a0 2024-01-21T19:57:37.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.682+0000 7fcaba52a700 1 --2- 172.21.15.173:0/537994679 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcab40f62a0 0x7fcab41086b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fcab0013510 tx=0x7fcab000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:37.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.682+0000 7fcaa6ffd700 1 -- 172.21.15.173:0/537994679 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcab0009320 con 0x7fcab40f62a0 2024-01-21T19:57:37.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.682+0000 7fcabb52c700 1 -- 172.21.15.173:0/537994679 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab4108da0 con 0x7fcab40f62a0 2024-01-21T19:57:37.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.682+0000 7fcaa6ffd700 1 -- 172.21.15.173:0/537994679 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcab0013f00 con 0x7fcab40f62a0 2024-01-21T19:57:37.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.682+0000 7fcaa6ffd700 1 -- 172.21.15.173:0/537994679 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcab0014dc0 con 0x7fcab40f62a0 2024-01-21T19:57:37.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.682+0000 7fcabb52c700 1 -- 172.21.15.173:0/537994679 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab4109230 con 0x7fcab40f62a0 2024-01-21T19:57:37.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.683+0000 7fcaa6ffd700 1 -- 172.21.15.173:0/537994679 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcab00095f0 con 0x7fcab40f62a0 2024-01-21T19:57:37.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.683+0000 7fcabb52c700 1 -- 172.21.15.173:0/537994679 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab4045ad0 con 0x7fcab40f62a0 2024-01-21T19:57:37.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.683+0000 7fcaa6ffd700 1 --2- 172.21.15.173:0/537994679 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fcaa002ff60 0x7fcaa0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:37.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.683+0000 7fcaa6ffd700 1 -- 172.21.15.173:0/537994679 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcab0041020 con 0x7fcab40f62a0 2024-01-21T19:57:37.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.684+0000 7fcab9d29700 1 --2- 172.21.15.173:0/537994679 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fcaa002ff60 0x7fcaa0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:37.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.684+0000 7fcab9d29700 1 --2- 172.21.15.173:0/537994679 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fcaa002ff60 0x7fcaa0032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fcaa80096a0 tx=0x7fcaa8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:37.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.688+0000 7fcaa6ffd700 1 -- 172.21.15.173:0/537994679 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcab0014420 con 0x7fcab40f62a0 2024-01-21T19:57:37.883 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.892+0000 7fcabb52c700 1 -- 172.21.15.173:0/537994679 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcab410a080 con 0x7fcab40f62a0 2024-01-21T19:57:37.883 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.892+0000 7fcaa6ffd700 1 -- 172.21.15.173:0/537994679 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcab001c920 con 0x7fcab40f62a0 2024-01-21T19:57:37.883 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:57:37.884 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:57:37.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.894+0000 7fcabb52c700 1 -- 172.21.15.173:0/537994679 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fcaa002ff60 msgr2=0x7fcaa0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:37.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.894+0000 7fcabb52c700 1 --2- 172.21.15.173:0/537994679 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fcaa002ff60 0x7fcaa0032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fcaa80096a0 tx=0x7fcaa8006b40).stop 2024-01-21T19:57:37.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.894+0000 7fcabb52c700 1 -- 172.21.15.173:0/537994679 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcab40f62a0 msgr2=0x7fcab41086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:37.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.894+0000 7fcabb52c700 1 --2- 172.21.15.173:0/537994679 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcab40f62a0 0x7fcab41086b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fcab0013510 tx=0x7fcab000b040).stop 2024-01-21T19:57:37.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.894+0000 7fcabb52c700 1 -- 172.21.15.173:0/537994679 shutdown_connections 2024-01-21T19:57:37.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.894+0000 7fcabb52c700 1 --2- 172.21.15.173:0/537994679 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fcaa002ff60 0x7fcaa0032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:37.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.894+0000 7fcabb52c700 1 --2- 172.21.15.173:0/537994679 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcab40f62a0 0x7fcab41086b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:37.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.894+0000 7fcabb52c700 1 -- 172.21.15.173:0/537994679 >> 172.21.15.173:0/537994679 conn(0x7fcab40f1aa0 msgr2=0x7fcab40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:37.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.894+0000 7fcabb52c700 1 -- 172.21.15.173:0/537994679 shutdown_connections 2024-01-21T19:57:37.886 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:37.894+0000 7fcabb52c700 1 -- 172.21.15.173:0/537994679 wait complete. 2024-01-21T19:57:37.886 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:57:39.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:57:37.020744+0000 mgr.smithi169.espgue (mgr.14162) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:57:39.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:38 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:57:37.893163+0000 mon.smithi169 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.173:0/537994679' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:57:39.287 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T19:57:39.287 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df2470f2-b895-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T19:57:40.541 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.548+0000 7fabd6453700 1 -- 172.21.15.173:0/3383513359 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabd00f5410 msgr2=0x7fabd00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:40.541 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.548+0000 7fabd6453700 1 --2- 172.21.15.173:0/3383513359 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabd00f5410 0x7fabd00f5830 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fabc0009fb0 tx=0x7fabc0009b30).stop 2024-01-21T19:57:40.541 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.548+0000 7fabd6453700 1 -- 172.21.15.173:0/3383513359 shutdown_connections 2024-01-21T19:57:40.541 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.548+0000 7fabd6453700 1 --2- 172.21.15.173:0/3383513359 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabd00f5410 0x7fabd00f5830 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:40.541 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.548+0000 7fabd6453700 1 -- 172.21.15.173:0/3383513359 >> 172.21.15.173:0/3383513359 conn(0x7fabd00f0b70 msgr2=0x7fabd00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:40.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.548+0000 7fabd6453700 1 -- 172.21.15.173:0/3383513359 shutdown_connections 2024-01-21T19:57:40.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.548+0000 7fabd6453700 1 -- 172.21.15.173:0/3383513359 wait complete. 2024-01-21T19:57:40.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.549+0000 7fabd6453700 1 Processor -- start 2024-01-21T19:57:40.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.549+0000 7fabd6453700 1 -- start start 2024-01-21T19:57:40.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.549+0000 7fabd6453700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabd00f5410 0x7fabd0109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:40.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.549+0000 7fabd6453700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd0060240 con 0x7fabd00f5410 2024-01-21T19:57:40.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.550+0000 7fabcffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabd00f5410 0x7fabd0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:40.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.550+0000 7fabcffff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabd00f5410 0x7fabd0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.173:56146/0 (socket says 172.21.15.173:56146) 2024-01-21T19:57:40.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.550+0000 7fabcffff700 1 -- 172.21.15.173:0/3356211620 learned_addr learned my addr 172.21.15.173:0/3356211620 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-01-21T19:57:40.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.550+0000 7fabcffff700 1 -- 172.21.15.173:0/3356211620 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc0005040 con 0x7fabd00f5410 2024-01-21T19:57:40.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.550+0000 7fabcffff700 1 --2- 172.21.15.173:0/3356211620 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabd00f5410 0x7fabd0109430 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fabc0013510 tx=0x7fabc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:40.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.550+0000 7fabccff9700 1 -- 172.21.15.173:0/3356211620 <== mon.0 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabc0009320 con 0x7fabd00f5410 2024-01-21T19:57:40.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.550+0000 7fabd6453700 1 -- 172.21.15.173:0/3356211620 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd0060440 con 0x7fabd00f5410 2024-01-21T19:57:40.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.550+0000 7fabccff9700 1 -- 172.21.15.173:0/3356211620 <== mon.0 v2:172.21.15.169:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fabc0013f00 con 0x7fabd00f5410 2024-01-21T19:57:40.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.550+0000 7fabccff9700 1 -- 172.21.15.173:0/3356211620 <== mon.0 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabc0014d90 con 0x7fabd00f5410 2024-01-21T19:57:40.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.550+0000 7fabd6453700 1 -- 172.21.15.173:0/3356211620 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd0109ae0 con 0x7fabd00f5410 2024-01-21T19:57:40.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.551+0000 7fabccff9700 1 -- 172.21.15.173:0/3356211620 <== mon.0 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fabc00263a0 con 0x7fabd00f5410 2024-01-21T19:57:40.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.551+0000 7fabccff9700 1 --2- 172.21.15.173:0/3356211620 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fabb802ff60 0x7fabb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:57:40.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.551+0000 7fabccff9700 1 -- 172.21.15.173:0/3356211620 <== mon.0 v2:172.21.15.169:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fabc0041000 con 0x7fabd00f5410 2024-01-21T19:57:40.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.551+0000 7fabd6453700 1 -- 172.21.15.173:0/3356211620 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd0059100 con 0x7fabd00f5410 2024-01-21T19:57:40.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.551+0000 7fabcf7fe700 1 --2- 172.21.15.173:0/3356211620 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fabb802ff60 0x7fabb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:57:40.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.552+0000 7fabcf7fe700 1 --2- 172.21.15.173:0/3356211620 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fabb802ff60 0x7fabb8032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fabc40096a0 tx=0x7fabc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:57:40.547 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.556+0000 7fabccff9700 1 -- 172.21.15.173:0/3356211620 <== mon.0 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fabc0012920 con 0x7fabd00f5410 2024-01-21T19:57:40.753 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.762+0000 7fabd6453700 1 -- 172.21.15.173:0/3356211620 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fabd0045ad0 con 0x7fabd00f5410 2024-01-21T19:57:40.754 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.763+0000 7fabccff9700 1 -- 172.21.15.173:0/3356211620 <== mon.0 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fabc002b020 con 0x7fabd00f5410 2024-01-21T19:57:40.754 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-01-21T19:57:40.754 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"df2470f2-b895-11ee-95b0-87774f69a715","modified":"2024-01-21T19:47:37.438649Z","created":"2024-01-21T19:47:37.438649Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T19:57:40.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.764+0000 7fabd6453700 1 -- 172.21.15.173:0/3356211620 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fabb802ff60 msgr2=0x7fabb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:40.756 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.764+0000 7fabd6453700 1 --2- 172.21.15.173:0/3356211620 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fabb802ff60 0x7fabb8032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fabc40096a0 tx=0x7fabc4006b40).stop 2024-01-21T19:57:40.756 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.764+0000 7fabd6453700 1 -- 172.21.15.173:0/3356211620 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabd00f5410 msgr2=0x7fabd0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:57:40.756 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.764+0000 7fabd6453700 1 --2- 172.21.15.173:0/3356211620 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabd00f5410 0x7fabd0109430 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fabc0013510 tx=0x7fabc000b040).stop 2024-01-21T19:57:40.756 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.764+0000 7fabd6453700 1 -- 172.21.15.173:0/3356211620 shutdown_connections 2024-01-21T19:57:40.756 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.764+0000 7fabd6453700 1 --2- 172.21.15.173:0/3356211620 >> [v2:172.21.15.169:6800/1381810408,v1:172.21.15.169:6801/1381810408] conn(0x7fabb802ff60 0x7fabb8032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:40.756 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.764+0000 7fabd6453700 1 --2- 172.21.15.173:0/3356211620 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fabd00f5410 0x7fabd0109430 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:57:40.756 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.764+0000 7fabd6453700 1 -- 172.21.15.173:0/3356211620 >> 172.21.15.173:0/3356211620 conn(0x7fabd00f0b70 msgr2=0x7fabd00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:57:40.756 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.765+0000 7fabd6453700 1 -- 172.21.15.173:0/3356211620 shutdown_connections 2024-01-21T19:57:40.756 INFO:teuthology.orchestra.run.smithi173.stderr:2024-01-21T19:57:40.765+0000 7fabd6453700 1 -- 172.21.15.173:0/3356211620 wait complete. 2024-01-21T19:57:40.756 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-01-21T19:57:41.148 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-01-21T19:57:41.150 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-21T19:57:41.150 DEBUG:teuthology.orchestra.run.smithi169:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-21T19:57:41.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:40 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: cluster 2024-01-21T19:57:39.021109+0000 mgr.smithi169.espgue (mgr.14162) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T19:57:41.164 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:40 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: audit 2024-01-21T19:57:40.763688+0000 mon.smithi169 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.173:0/3356211620' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T19:57:41.174 DEBUG:teuthology.orchestra.run.smithi173:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-21T19:57:41.191 INFO:tasks.cephadm:Stopping all daemons... 2024-01-21T19:57:41.191 INFO:tasks.cephadm.mon.smithi169:Stopping mon.smithi169... 2024-01-21T19:57:41.192 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-df2470f2-b895-11ee-95b0-87774f69a715@mon.smithi169 2024-01-21T19:57:41.508 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:41 smithi169 systemd[1]: Stopping Ceph mon.smithi169 for df2470f2-b895-11ee-95b0-87774f69a715... 2024-01-21T19:57:41.508 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:41 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:57:41.331+0000 7f8811978700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi169 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-21T19:57:41.508 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Jan 21 19:57:41 smithi169 ceph-df2470f2-b895-11ee-95b0-87774f69a715-mon.smithi169[38820]: debug 2024-01-21T19:57:41.331+0000 7f8811978700 -1 mon.smithi169@0(leader) e1 *** Got Signal Terminated *** 2024-01-21T19:57:41.655 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df2470f2-b895-11ee-95b0-87774f69a715@mon.smithi169.service' 2024-01-21T19:57:41.701 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-21T19:57:41.702 INFO:tasks.cephadm.mon.smithi169:Stopped mon.smithi169 2024-01-21T19:57:41.702 INFO:tasks.cephadm.mon.smithi173:Stopping mon.smithi173... 2024-01-21T19:57:41.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-df2470f2-b895-11ee-95b0-87774f69a715@mon.smithi173 2024-01-21T19:57:41.735 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ceph-df2470f2-b895-11ee-95b0-87774f69a715@mon.smithi173.service: Unit ceph-df2470f2-b895-11ee-95b0-87774f69a715@mon.smithi173.service not loaded. 2024-01-21T19:57:41.736 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-01-21T19:57:41.736 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi173" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi173 with status 5: 'sudo systemctl stop ceph-df2470f2-b895-11ee-95b0-87774f69a715@mon.smithi173' 2024-01-21T19:57:41.737 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-21T19:57:41.739 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525059/remote/ubuntu@smithi169.front.sepia.ceph.com/crash 2024-01-21T19:57:41.740 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/crash -- . 2024-01-21T19:57:41.774 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525059/remote/ubuntu@smithi173.front.sepia.ceph.com/crash 2024-01-21T19:57:41.774 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/crash -- . 2024-01-21T19:57:41.816 INFO:teuthology.orchestra.run.smithi173.stderr:tar: /var/lib/ceph/df2470f2-b895-11ee-95b0-87774f69a715/crash: Cannot open: No such file or directory 2024-01-21T19:57:41.816 INFO:teuthology.orchestra.run.smithi173.stderr:tar: Error is not recoverable: exiting now 2024-01-21T19:57:41.817 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-21T19:57:41.817 DEBUG:teuthology.orchestra.run.smithi169:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-21T19:57:41.846 INFO:teuthology.orchestra.run.smithi169.stderr:grep: /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715/ceph.log: No such file or directory 2024-01-21T19:57:41.848 INFO:tasks.cephadm:Compressing logs... 2024-01-21T19:57:41.848 DEBUG:teuthology.orchestra.run.smithi169:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-21T19:57:41.890 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-21T19:57:41.916 INFO:teuthology.orchestra.run.smithi169.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-01-21T19:57:41.916 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-21T19:57:41.916 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715/ceph-volume.log 2024-01-21T19:57:41.916 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715/ceph-mon.smithi169.log 2024-01-21T19:57:41.916 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715/ceph-mgr.smithi169.espgue.log 2024-01-21T19:57:41.917 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715/ceph-volume.log: /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715/ceph-mon.smithi169.log: /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715/ceph-mgr.smithi169.espgue.log: 90.8% -- replaced with /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715/ceph-volume.log.gz 2024-01-21T19:57:41.918 INFO:teuthology.orchestra.run.smithi173.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-21T19:57:41.918 INFO:teuthology.orchestra.run.smithi173.stderr:: No such file or directory 2024-01-21T19:57:41.920 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cephadm.log: 96.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-21T19:57:41.921 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-01-21T19:57:41.921 INFO:teuthology.orchestra.run.smithi173.stderr:real 0m0.013s 2024-01-21T19:57:41.921 INFO:teuthology.orchestra.run.smithi173.stderr:user 0m0.010s 2024-01-21T19:57:41.921 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m0.011s 2024-01-21T19:57:41.922 INFO:teuthology.orchestra.run.smithi169.stderr: 92.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-21T19:57:41.993 INFO:teuthology.orchestra.run.smithi169.stderr:gzip: /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715/ceph-mgr.smithi169.espgue.log: file size changed while zipping 2024-01-21T19:57:41.993 INFO:teuthology.orchestra.run.smithi169.stderr: 92.6% -- replaced with /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715/ceph-mgr.smithi169.espgue.log.gz 2024-01-21T19:57:42.059 INFO:teuthology.orchestra.run.smithi169.stderr: 90.5% -- replaced with /var/log/ceph/df2470f2-b895-11ee-95b0-87774f69a715/ceph-mon.smithi169.log.gz 2024-01-21T19:57:42.063 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-01-21T19:57:42.063 INFO:teuthology.orchestra.run.smithi169.stderr:real 0m0.155s 2024-01-21T19:57:42.063 INFO:teuthology.orchestra.run.smithi169.stderr:user 0m0.231s 2024-01-21T19:57:42.063 INFO:teuthology.orchestra.run.smithi169.stderr:sys 0m0.017s 2024-01-21T19:57:42.063 INFO:tasks.cephadm:Archiving logs... 2024-01-21T19:57:42.063 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525059/remote/ubuntu@smithi169.front.sepia.ceph.com/log 2024-01-21T19:57:42.065 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-21T19:57:42.174 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525059/remote/ubuntu@smithi173.front.sepia.ceph.com/log 2024-01-21T19:57:42.175 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-21T19:57:42.204 INFO:tasks.cephadm:Removing cluster... 2024-01-21T19:57:42.204 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid df2470f2-b895-11ee-95b0-87774f69a715 --force 2024-01-21T19:58:25.313 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid df2470f2-b895-11ee-95b0-87774f69a715 --force 2024-01-21T19:58:25.623 INFO:tasks.cephadm:Removing cephadm ... 2024-01-21T19:58:25.624 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-21T19:58:25.643 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-21T19:58:25.660 INFO:tasks.cephadm:Teardown complete 2024-01-21T19:58:25.661 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi173 with status 5: 'sudo systemctl stop ceph-df2470f2-b895-11ee-95b0-87774f69a715@mon.smithi173' 2024-01-21T19:58:25.865 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b52dded03dcf4ecaa529652c05d32661 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi173 with status 5: 'sudo systemctl stop ceph-df2470f2-b895-11ee-95b0-87774f69a715@mon.smithi173' 2024-01-21T19:58:25.870 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-21T19:58:25.879 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-21T19:58:25.893 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-21T19:58:25.893 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-21T19:58:25.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-21T19:58:25.981 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-21T19:58:25.992 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-21T19:58:25.992 DEBUG:teuthology.orchestra.run.smithi169:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T19:58:25.995 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T19:58:26.010 INFO:teuthology.orchestra.run.smithi169.stderr:bash: ntpq: command not found 2024-01-21T19:58:26.011 INFO:teuthology.orchestra.run.smithi173.stderr:bash: ntpq: command not found 2024-01-21T19:58:26.014 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T19:58:26.014 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-01-21T19:58:26.014 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 24 -334us[ -302us] +/- 67ms 2024-01-21T19:58:26.014 INFO:teuthology.orchestra.run.smithi169.stdout:^* hv02.front.sepia.ceph.com 3 6 377 20 -680us[ -648us] +/- 69ms 2024-01-21T19:58:26.015 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 +1315us[+1348us] +/- 87ms 2024-01-21T19:58:26.015 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:58:26.016 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T19:58:26.016 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-01-21T19:58:26.017 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 -350us[ -357us] +/- 67ms 2024-01-21T19:58:26.017 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv02.front.sepia.ceph.com 3 6 377 22 -694us[ -701us] +/- 69ms 2024-01-21T19:58:26.017 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 +1289us[+1289us] +/- 87ms 2024-01-21T19:58:26.017 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:58:26.017 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-21T19:58:26.028 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-21T19:58:26.029 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-21T19:58:26.042 DEBUG:teuthology.orchestra.run.smithi169:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-21T19:58:26.116 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-21T19:58:26.179 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T19:58:26.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T19:58:26.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T19:58:26.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T19:58:26.239 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-21T19:58:26.248 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-21T19:58:26.262 INFO:teuthology.task.internal:Duration was 1206.823622 seconds 2024-01-21T19:58:26.262 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-21T19:58:26.306 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-21T19:58:26.306 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-21T19:58:26.309 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-21T19:58:26.354 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T19:58:26.354 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T19:58:26.679 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-21T19:58:26.679 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi169.front.sepia.ceph.com 2024-01-21T19:58:26.679 DEBUG:teuthology.orchestra.run.smithi169:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-21T19:58:26.701 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-01-21T19:58:26.701 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-21T19:58:26.723 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-21T19:58:26.724 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-21T19:58:26.743 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-21T19:58:26.812 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-21T19:58:26.812 DEBUG:teuthology.orchestra.run.smithi169:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-21T19:58:26.814 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-21T19:58:26.902 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-21T19:58:26.911 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-21T19:58:26.911 DEBUG:teuthology.orchestra.run.smithi169:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-21T19:58:26.938 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-21T19:58:26.965 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-21T19:58:26.977 DEBUG:teuthology.orchestra.run.smithi169:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-21T19:58:26.981 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-21T19:58:27.006 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = core 2024-01-21T19:58:27.032 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-01-21T19:58:27.057 DEBUG:teuthology.orchestra.run.smithi169:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-21T19:58:27.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T19:58:27.096 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-21T19:58:27.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T19:58:27.114 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-21T19:58:27.126 INFO:teuthology.task.internal:Transferring archived files... 2024-01-21T19:58:27.126 DEBUG:teuthology.misc:Transferring archived files from smithi169:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525059/remote/smithi169 2024-01-21T19:58:27.127 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-21T19:58:27.175 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525059/remote/smithi173 2024-01-21T19:58:27.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-21T19:58:27.215 INFO:teuthology.task.internal:Removing archive directory... 2024-01-21T19:58:27.215 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-21T19:58:27.217 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-21T19:58:27.266 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-21T19:58:27.277 INFO:teuthology.task.internal:Not uploading archives. 2024-01-21T19:58:27.277 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-21T19:58:27.291 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-21T19:58:27.291 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-21T19:58:27.293 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-21T19:58:27.308 INFO:teuthology.orchestra.run.smithi169.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 21 19:58 /home/ubuntu/cephtest 2024-01-21T19:58:27.324 INFO:teuthology.orchestra.run.smithi173.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 21 19:58 /home/ubuntu/cephtest 2024-01-21T19:58:27.325 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-21T19:58:27.336 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-21T19:58:27.358 INFO:teuthology.nuke:Checking targets against current locks 2024-01-21T19:58:27.383 DEBUG:teuthology.nuke:shortname: smithi169 2024-01-21T19:58:27.383 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T19:58:27.407 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525059', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 19:29:25.919598', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPebK3S0muhRwMXStynRW0aoWwxHFF3sjrD/oODnl7qYpB7hdQsoZW1TKv1Ezb4tJZy16hfjIJBJUL6oPdvScpo='} 2024-01-21T19:58:27.411 DEBUG:teuthology.nuke:shortname: smithi173 2024-01-21T19:58:27.411 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T19:58:27.437 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525059', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 19:29:25.921557', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ3RlVg3usAaY8CQmD/14tkcGNhuOOBr183OX1FthtbBppgoH2/4EHU2ec/NFzXx791hRiW0c+lfxobzK60ZI2w='} 2024-01-21T19:58:27.468 INFO:teuthology.orchestra.console.smithi169:Power off 2024-01-21T19:58:27.469 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-21T19:58:27.505 INFO:teuthology.orchestra.console.smithi173:Power off 2024-01-21T19:58:27.506 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-21T19:58:27.515 DEBUG:teuthology.orchestra.console.smithi169:power off output: Chassis Power Control: Down/Off 2024-01-21T19:58:27.516 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T19:58:27.539 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-01-21T19:58:27.548 DEBUG:teuthology.orchestra.console.smithi173:power off output: Chassis Power Control: Down/Off 2024-01-21T19:58:27.548 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T19:58:27.574 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-01-21T19:58:31.539 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T19:58:31.576 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T19:58:31.650 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-01-21T19:58:31.686 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-01-21T19:58:35.652 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T19:58:35.687 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T19:58:35.762 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2024-01-21T19:58:35.796 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is off 2024-01-21T19:58:35.863 INFO:teuthology.orchestra.console.smithi169:Power off completed 2024-01-21T19:58:35.897 INFO:teuthology.orchestra.console.smithi173:Power off completed 2024-01-21T19:58:36.002 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1206.8236215114594 failure_reason: 'Command failed on smithi173 with status 5: ''sudo systemctl stop ceph-df2470f2-b895-11ee-95b0-87774f69a715@mon.smithi173''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=b52dded03dcf4ecaa529652c05d32661 status: fail success: false 2024-01-21T19:58:36.002 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-21T19:58:36.103 INFO:teuthology.run:FAIL